Remove Komodo backend in favor of native hosts

This commit is contained in:
2026-10-05 18:07:07 -04:00
parent 701066aaf8
commit 4b0f341d64
116 changed files with 557 additions and 8132 deletions
-2
View File
@@ -28,8 +28,6 @@ PLUTAINER_IMAGE=ghcr.io/ayymoss/plutainer:latest
# Browser login on localhost only (never for an exposed controller): # Browser login on localhost only (never for an exposed controller):
# EC_AUTH_ALLOW_HTTP=true # EC_AUTH_ALLOW_HTTP=true
# Existing legacy registrations can still use their previous KOMODO_*,
# SERVERS_ROOT, T6_SERVERS_ROOT, and RCON_HOST settings. Native hosts need none.
# Fleet concurrency and campaign retention are saved in Settings, not env vars. # Fleet concurrency and campaign retention are saved in Settings, not env vars.
# Optional request/report timings and event-loop delay diagnostics. # Optional request/report timings and event-loop delay diagnostics.
-4
View File
@@ -3,7 +3,6 @@
Electric Cherry manages T5 and T6 Plutonium Zombies servers through its own host Electric Cherry manages T5 and T6 Plutonium Zombies servers through its own host
agents. Docker and Plutainer run the games; EC handles hosts, game files, agents. Docker and Plutainer run the games; EC handles hosts, game files,
collections, configuration, campaigns, maintenance and operation history. collections, configuration, campaigns, maintenance and operation history.
Komodo is not required for native hosts.
## Getting started ## Getting started
@@ -134,9 +133,6 @@ affect occupancy or game history. Existing JSON payloads are unchanged.
- [Metrics](docs/metrics.md) - [Metrics](docs/metrics.md)
- [Leaderboard and match history](docs/leaderboard.md) - [Leaderboard and match history](docs/leaderboard.md)
Older workflow documents may describe the legacy Komodo backend. Native hosts
use the agent boundary described in the native-host document.
## Development ## Development
The controller is SvelteKit/TypeScript with SQLite via better-sqlite3, the Node The controller is SvelteKit/TypeScript with SQLite via better-sqlite3, the Node
+6 -5
View File
@@ -104,7 +104,8 @@ the current shortfall too.
All collections and individual launches share one startup queue, ordered by request All collections and individual launches share one startup queue, ordered by request
time. By default, one server starts at a time. Its startup slot remains occupied time. By default, one server starts at a time. Its startup slot remains occupied
until Komodo is healthy, RCON answers, and the reported map matches the selection. until the host agent confirms container health, game status and the selected map
are verified, and RCON reports a current Plutonium revision.
Only then can the next queued server start. Set **Concurrent server startups** in **Settings → Fleet settings** to Only then can the next queued server start. Set **Concurrent server startups** in **Settings → Fleet settings** to
a positive integer to allow more simultaneous startups. Each worker pass admits a positive integer to allow more simultaneous startups. Each worker pass admits
at most the available slots, even if an attempt fails immediately. at most the available slots, even if an attempt fails immediately.
@@ -137,10 +138,10 @@ another launch takes the lobby, submission stops before saving the key;
refresh the preview and review the newly assigned ID. Refreshing a preview clears refresh the preview and review the newly assigned ID. Refreshing a preview clears
the typed key so it cannot accidentally be saved against a different lobby. the typed key so it cannot accidentally be saved against a different lobby.
Komodo connection settings, target host, RCON configuration, config filename, Host selection, game-file versions, runtime configuration and lobby availability
lobby availability, and port availability are checked locally. These checks do are checked before queueing. Provisioning assigns host capacity and physical
not probe the host or verify game files, images, Docker volumes, firewall ports when dispatching. The host agent performs container and RCON verification
rules, or credentials. Container and RCON verification happen after provisioning. locally after launch. See [native hosts](native-hosts.md) for readiness checks.
T6 variants retain their exact catalog slug and launch ruleset; RCON health checks T6 variants retain their exact catalog slug and launch ruleset; RCON health checks
verify the engine map, not the variant's start location. verify the engine map, not the variant's start location.
+1 -1
View File
@@ -122,7 +122,7 @@ an application restart, routine watchdog checks wait for a new status-monitor pa
Provisioning retains its checks before executing recovery actions. Provisioning retains its checks before executing recovery actions.
Three observations over at least two minutes are required before an automatic Three observations over at least two minutes are required before an automatic
health restart; starting or restarting containers get five minutes. Unknown Komodo responses are observations, health restart; starting or restarting containers get five minutes. Unknown host-agent responses are observations,
not proof of a stopped container. Repeated manual checks do not accelerate this not proof of a stopped container. Repeated manual checks do not accelerate this
window. A successful routine or manual game-status probe between scans resets window. A successful routine or manual game-status probe between scans resets
the failure streak. Missing RCON configuration, unresolved ports, and invalid the failure streak. Missing RCON configuration, unresolved ports, and invalid
+3 -8
View File
@@ -1,8 +1,7 @@
# Native host management # Native host management
Electric Cherry controls Docker/Plutainer through an outbound host agent. A native Electric Cherry controls Docker/Plutainer through an outbound host agent. The controller
instance never uses Komodo or a filesystem mounted into the controller. The does not need mounted game-server files. It owns desired state, campaigns, collections, safety queues and history;
controller owns desired state, campaigns, collections, safety queues and history;
the agent owns local execution and its durable operation journal. the agent owns local execution and its durable operation journal.
Fleet health checks query RCON even when container inspection is unavailable. A Fleet health checks query RCON even when container inspection is unavailable. A
@@ -554,16 +553,12 @@ coverage is not evidence that a real T5/T6 server has been deployed here. Run:
execution and after execution but before acknowledgement. Verify the same execution and after execution but before acknowledgement. Verify the same
operation IDs and absence of duplicate containers/restarts. operation IDs and absence of duplicate containers/restarts.
Legacy backend code remains for existing database registrations. New native
launches do not require its configuration, API, network or mounted directories.
### Provisioning progress and diagnostics ### Provisioning progress and diagnostics
Native launches distinguish initial eligibility evaluation from an actual wait Native launches distinguish initial eligibility evaluation from an actual wait
for startup capacity. The operation page records payload preparation, command for startup capacity. The operation page records payload preparation, command
submission, agent acknowledgement and execution, and separate container, game, submission, agent acknowledgement and execution, and separate container, game,
map and Plutonium revision checks. Native actions use host-agent steps rather map and Plutonium revision checks. Evidence remains available after completion.
than legacy Komodo deployment steps. Evidence remains available after completion.
The **Host agent execution** panel reads the durable command state directly, The **Host agent execution** panel reads the durable command state directly,
including its queue time and latest report time. Agent receipts are recorded in including its queue time and latest report time. Agent receipts are recorded in
+2 -2
View File
@@ -111,7 +111,7 @@ Separate durable milestones from transient readiness checks. A passed container/
| --- | --- | | --- | --- |
| `canCancelRemoval` | Offer **Cancel queued removal**, explaining that it is available only before removal begins. Preserve the existing atomic worker/cancellation race protection. | | `canCancelRemoval` | Offer **Cancel queued removal**, explaining that it is available only before removal begins. Preserve the existing atomic worker/cancellation race protection. |
| `canResumeVerification` | Offer **Resume health verification** and explain that it observes the existing instance without creating or restarting it. | | `canResumeVerification` | Offer **Resume health verification** and explain that it observes the existing instance without creating or restarting it. |
| `canResumeCleanup` | Offer **Resume cleanup**, including eligible deleted identities. Explain that Komodo deletion already succeeded. | | `canResumeCleanup` | Offer **Resume cleanup**, including eligible deleted identities. Explain that instance removal already succeeded. |
| `canResumeRebuild` | Offer **Resume rebuild cleanup** and explain that local cleanup precedes creation in the reserved slot. | | `canResumeRebuild` | Offer **Resume rebuild cleanup** and explain that local cleanup precedes creation in the reserved slot. |
| Failed/interrupted without a resume capability | Lead with the recorded failure and relevant inspection destination. An uncertain remote response must not produce a generic retry command. | | Failed/interrupted without a resume capability | Lead with the recorded failure and relevant inspection destination. An uncertain remote response must not produce a generic retry command. |
| Cancelled | State that the request is cancelled; distinguish recorded prior work from remaining work that did not run. Do not claim continued execution. | | Cancelled | State that the request is cancelled; distinguish recorded prior work from remaining work that did not run. Do not claim continued execution. |
@@ -154,7 +154,7 @@ A suitable small schema extension is nullable `phase_key`, `phase_started_at`, `
Implementation rules: Implementation rules:
1. Record structured blockers and observations where the worker already makes decisions. Do not build a second eligibility engine in the API or browser. The detail GET remains read-only and performs no Komodo/RCON probes. 1. Record structured blockers and observations where the worker already makes decisions. Do not build a second eligibility engine in the API or browser. The detail GET remains read-only and performs no host-agent/RCON probes.
2. Derive current capacity from existing scheduler helpers/tables, with an explicit snapshot time. Persist important milestones against the operation because server-owned recovery rows can later be replaced or deleted. Only attach a live slot when its `operation_id` matches; slot absence alone is not evidence of successful release. 2. Derive current capacity from existing scheduler helpers/tables, with an explicit snapshot time. Persist important milestones against the operation because server-owned recovery rows can later be replaced or deleted. Only attach a live slot when its `operation_id` matches; slot absence alone is not evidence of successful release.
3. Update related state, observation, and semantic event atomically around each durable checkpoint, keeping network calls outside database transactions. Dual-write existing stage/event text, especially `REMOVING_SERVER`, which participates in recovery eligibility today. 3. Update related state, observation, and semantic event atomically around each durable checkpoint, keeping network calls outside database transactions. Dual-write existing stage/event text, especially `REMOVING_SERVER`, which participates in recovery eligibility today.
4. Record meaningful phase, blocker, command, and recovery transitions. Update observation timestamps on repeated checks without appending an event every poll. Changing player counts must not reset the phase's start time. 4. Record meaningful phase, blocker, command, and recovery transitions. Update observation timestamps on repeated checks without appending an event every poll. Changing player counts must not reset the phase's start time.
+16 -20
View File
@@ -12,9 +12,9 @@ server, choose **Manage server** or open its **Server actions** menu.
| Create | Create a new instance using the latest promoted files and verify container health, game status, the selected map, and the Plutonium revision. | | Create | Create a new instance using the latest promoted files and verify container health, game status, the selected map, and the Plutonium revision. |
| Start | Start the existing container and verify container health and RCON. | | Start | Start the existing container and verify container health and RCON. |
| Restart | Wait until empty, restart the existing container, and verify container health and RCON. | | Restart | Wait until empty, restart the existing container, and verify container health and RCON. |
| Rebuild | Wait until empty, destroy the instance, delete all server files and its registration, then create a fresh instance in the same server slot and port. Verify container health, game status, map, and Plutonium revision. | | Rebuild | Wait until empty, replace the instance through the host agent, preserving its server identity and port while advancing its generation. Verify container health, game status, map, and Plutonium revision. |
| Stop | Wait until empty, stop the container, and verify that Komodo reports it stopped. Keep the instance for a later Start. | | Stop | Wait until empty, stop the container, and verify that the host agent reports it stopped. Keep the instance for a later Start. |
| Destroy | Wait until empty, remove the container and Komodo deployment, then delete server files and registration. | | Destroy | Wait until empty, remove the container and its files through the host agent, then retire the instance and registration. |
Servers linked to a running or ready-to-promote beta cannot be destroyed or Servers linked to a running or ready-to-promote beta cannot be destroyed or
rebuilt. Promote or roll back the beta first, including paused or failed test rebuilt. Promote or roll back the beta first, including paused or failed test
@@ -32,11 +32,10 @@ activity belong to Electric Cherry, and are not instance files.
Rebuild reserves the original ID and port throughout teardown and creation, even Rebuild reserves the original ID and port throughout teardown and creation, even
when a lower lobby is available. It shares the existing creation queue and startup when a lower lobby is available. It shares the existing creation queue and startup
limit. A failed removal never proceeds to creation. Once Komodo deletion is limit. The agent records each operation durably, and the controller reconciles its
confirmed, failed local cleanup offers **Resume rebuild cleanup**; creation cannot receipt after reconnecting or restarting. An interrupted fresh startup can resume
proceed until the old files are fully removed. An interrupted fresh startup can health verification without repeating the operation. Inspect failures before
resume health verification, without repeating teardown or creation. Inspect other making another request.
failures before making another request.
Create and Rebuild finish only after RCON reports a runtime revision at least as Create and Rebuild finish only after RCON reports a runtime revision at least as
new as the current Plutonium production release. The worker checks the production new as the current Plutonium production release. The worker checks the production
@@ -49,16 +48,15 @@ updater configuration and server logs before retrying. Telemetry can verify game
status and map, but cannot verify the Plutonium revision. status and map, but cannot verify the Plutonium revision.
Start and Restart share the rolling restart limit with campaigns, betas, and Start and Restart share the rolling restart limit with campaigns, betas, and
maintenance. A successful API reply is followed by a recovery delay and actual maintenance. A successful agent receipt is followed by actual container and game-status checks. Failed or ambiguous restart requests retain their recovery
container and game-status checks. Failed or ambiguous restart requests retain their recovery
slot and are not automatically replayed. Stop requires an explicit stopped container state; an slot and are not automatically replayed. Stop requires an explicit stopped container state; an
unreachable Komodo API is not evidence that a stop succeeded. unreachable host agent is not evidence that a stop succeeded.
Disruptive operations require a fresh zero-player result from RCON or accepted Disruptive operations require a fresh zero-player result from RCON or accepted
telemetry, unless Komodo positively reports a stopped container. Fresh occupied telemetry, unless the host agent positively reports a stopped container. Fresh occupied
telemetry vetoes a conflicting empty RCON reading. Reports must be received within telemetry vetoes a conflicting empty RCON reading. Reports must be received within
90 seconds; recovery verification also excludes reports from before the submitted 90 seconds; recovery verification also excludes reports from before the submitted
operation. Telemetry can verify the game's activity and selected map after Komodo operation. Telemetry can verify the game's activity and selected map after the host agent
reports healthy. Runtime actions wait for conflicting reports healthy. Runtime actions wait for conflicting
campaign, beta, maintenance, or provisioning work. Duplicate pending requests for campaign, beta, maintenance, or provisioning work. Duplicate pending requests for
the same action return the existing operation; a different action returns a the same action return the existing operation; a different action returns a
@@ -78,8 +76,7 @@ after five minutes; unrelated recovery slots remain held. Drift campaigns follow
their linked rebuild without requesting another restart. Their file evidence and their linked rebuild without requesting another restart. Their file evidence and
history remain available after the instance is removed. history remain available after the instance is removed.
Campaign changes queued before submission can share one restart. Changes arriving Each campaign activation has its own durable operation and file selection. A failed or interrupted request keeps
after submission require a new operation. A failed or interrupted request keeps
its original ID and never automatically replays. An explicit beta retry starts a its original ID and never automatically replays. An explicit beta retry starts a
new attempt, preserving prior history; paused or closed beta work cannot start a new attempt, preserving prior history; paused or closed beta work cannot start a
queued restart. Recovery of an already submitted restart continues even if its queued restart. Recovery of an already submitted restart continues even if its
@@ -177,7 +174,7 @@ heartbeat or completion-time promises.
literal-text `query`, `limit` (default 25, maximum 100), an exclusive event-ID literal-text `query`, `limit` (default 25, maximum 100), an exclusive event-ID
`cursor`, and an upper-ID `watermark`. Keep the returned watermark when loading `cursor`, and an upper-ID `watermark`. Keep the returned watermark when loading
more and use `nextCursor` until it is null. Begin without a watermark to include more and use `nextCursor` until it is null. Begin without a watermark to include
newly arrived events. These reads do not probe Komodo or RCON. newly arrived events. These reads do not contact host agents or probe RCON.
Old `/deployments` pages redirect to their `/provisioning` equivalents, preserving Old `/deployments` pages redirect to their `/provisioning` equivalents, preserving
query parameters. Collection launch links at `/provisioning/collections/:id` or query parameters. Collection launch links at `/provisioning/collections/:id` or
@@ -185,7 +182,6 @@ query parameters. Collection launch links at `/provisioning/collections/:id` or
query parameters. `/api/deployments` remains compatible, including old campaign query parameters. `/api/deployments` remains compatible, including old campaign
detail endpoints. No history or IDs are renamed or discarded. detail endpoints. No history or IDs are renamed or discarded.
Start and Stop use Komodo's corresponding runtime commands. Rebuild uses full Start and Stop use durable `instance.start` and `instance.stop` host-agent
destruction and creation, not Komodo's redeploy command. See the operations. Rebuild uses `instance.rebuild` and replaces the instance files.
[Komodo execute API](https://docs.rs/komodo_client/latest/komodo_client/api/execute/index.html) See [native host management](native-hosts.md) for command delivery and recovery.
for the underlying container operations.
+2 -2
View File
@@ -1,6 +1,6 @@
# Reducing idle CPU use in T5 and T6 Zombies servers # Reducing idle CPU use in T5 and T6 Zombies servers
Research date: September 19, 2026. Scope: Plutonium T5 Zombies (`t5sp`) and T6 Zombies (`t6zm`), with particular attention to Electric Cherry's Plutainer/Docker/Komodo deployment. Research date: September 19, 2026. Scope: Plutonium T5 Zombies (`t5sp`) and T6 Zombies (`t6zm`), with particular attention to Electric Cherry's Plutainer/Docker host deployment.
**Recommendation:** first benchmark Wine's NTSYNC support and audit the effective runtime configuration. If every lobby must remain immediately available, trial CPU limits that apply only while a server is confirmed empty. If spare lobbies can be unavailable temporarily, keep a smaller pool running and start additional servers when demand rises. Stopping unused instances offers the most predictable reduction in their CPU consumption. **Recommendation:** first benchmark Wine's NTSYNC support and audit the effective runtime configuration. If every lobby must remain immediately available, trial CPU limits that apply only while a server is confirmed empty. If spare lobbies can be unavailable temporarily, keep a smaller pool running and start additional servers when demand rises. Stopping unused instances offers the most predictable reduction in their CPU consumption.
@@ -60,7 +60,7 @@ The policy should handle several cases explicitly:
- Give startup, map changes, and recovery the normal CPU budget. - Give startup, map changes, and recovery the normal CPU budget.
- Restore that budget on a connection attempt if a reliable signal exists. Otherwise, use a conservative idle limit that still supports joining and a detection interval demonstrated to be fast enough. - Restore that budget on a connection attempt if a reliable signal exists. Otherwise, use a conservative idle limit that still supports joining and a detection interval demonstrated to be fast enough.
- Restore the active budget when occupancy becomes unknown. A separate watchdog or expiring idle-policy lease should restore it if the controller stops running. - Restore the active budget when occupancy becomes unknown. A separate watchdog or expiring idle-policy lease should restore it if the controller stops running.
- Coordinate quota ownership with Komodo so container recreation does not silently lose the policy or reinstate an obsolete limit. - Coordinate quota ownership with the host agent so container recreation does not silently lose the policy or reinstate an obsolete limit.
Electric Cherry's current 30-second status cycle can leave a newly joined player under the idle cap until the next successful poll. An aggressive cap should not depend on that cycle alone. Quotas can also allow brief bursts followed by throttling within their scheduling period, so evaluate join time and responsiveness, not just average CPU. The kernel exposes `nr_throttled` and `throttled_usec` in cgroup v2 `cpu.stat` to help identify this behavior. [Linux CPU controller](https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu). Electric Cherry's current 30-second status cycle can leave a newly joined player under the idle cap until the next successful poll. An aggressive cap should not depend on that cycle alone. Quotas can also allow brief bursts followed by throttling within their scheduling period, so evaluate join time and responsiveness, not just average CPU. The kernel exposes `nr_throttled` and `throttled_usec` in cgroup v2 `cpu.stat` to help identify this behavior. [Linux CPU controller](https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu).
+2 -2
View File
@@ -44,7 +44,7 @@ Activity and notifications have been removed. Copy results appear next to their
- Promotion uses `GET /api/betas/:id/promote` to review impact and `POST` with the returned token as `previewToken` to apply it. A changed plan returns HTTP 409 and needs renewed review. - Promotion uses `GET /api/betas/:id/promote` to review impact and `POST` with the returned token as `previewToken` to apply it. A changed plan returns HTTP 409 and needs renewed review.
- Reconciliation uses `POST /api/drift` with `dryRun: true` and optional `servers`, then executes with the returned `expectedFingerprint`. Omitted servers explicitly mean the entire fleet. A stale review returns HTTP 409; changes detected during application are reported as skipped files. - Reconciliation uses `POST /api/drift` with `dryRun: true` and optional `servers`, then executes with the returned `expectedFingerprint`. Omitted servers explicitly mean the entire fleet. A stale review returns HTTP 409; changes detected during application are reported as skipped files.
- `POST /api/deployments/:id/cancel` cancels remaining provisioning work, including restarts and health checks. It does not undo changes already made. - `POST /api/deployments/:id/cancel` cancels remaining provisioning work, including restarts and health checks. It does not undo changes already made.
- System status exposes no credential values. Its optional connection check verifies Komodo network reachability, not credential validity. Remote game-file volumes remain explicitly unverified. - System status exposes no credential values. It reports host-agent readiness and verified game-file caches.
- Drafts contain nonsecret beta setup. Launch keys are not stored in browser drafts. - Drafts contain nonsecret beta setup. Launch keys are not stored in browser drafts.
- Log search/export covers the loaded tail, not an entire remote file or a full diagnostic archive. - Log search/export covers the loaded tail, not an entire remote file or a full diagnostic archive.
@@ -56,4 +56,4 @@ Activity and notifications have been removed. Copy results appear next to their
- Local browser walkthroughs use an isolated database and synthetic populated API fixtures, with production integration credentials disabled. Fleet, server diagnostics, logs, Settings, Leaderboard, libraries, beta setup/progress, launches, keys, and reconciliation are exercised at desktop and phone widths. - Local browser walkthroughs use an isolated database and synthetic populated API fixtures, with production integration credentials disabled. Fleet, server diagnostics, logs, Settings, Leaderboard, libraries, beta setup/progress, launches, keys, and reconciliation are exercised at desktop and phone widths.
- Browser checks include the loading button accessibility tree, clickable native labels, URL/Back restoration, retained previews/drafts, selected-tab visibility, pending import ownership, and horizontal overflow. These are targeted checks, not a claim of complete accessibility conformance. - Browser checks include the loading button accessibility tree, clickable native labels, URL/Back restoration, retained previews/drafts, selected-tab visibility, pending import ownership, and horizontal overflow. These are targeted checks, not a claim of complete accessibility conformance.
Live Komodo provisioning, real game-server restarts and removals were not executed during verification. Their boundary behavior is covered by regression tests and simulated UI flows. Live host provisioning, real game-server restarts and removals were not executed during verification. Their boundary behavior is covered by regression tests and simulated UI flows.
+3 -3
View File
@@ -10,7 +10,7 @@ Electric Cherry has a coherent visual foundation and unusually capable operation
This review combines source inspection with a local browser walkthrough. It covers Fleet, server details and diagnostics, Scripts, Configs, Plugins, beta creation and progress, server deployments, blueprints, campaigns, keys, and Leaderboard. Existing design notes and workflow documentation informed the review. This review combines source inspection with a local browser walkthrough. It covers Fleet, server details and diagnostics, Scripts, Configs, Plugins, beta creation and progress, server deployments, blueprints, campaigns, keys, and Leaderboard. Existing design notes and workflow documentation informed the review.
Browser exploration used an isolated database with Komodo/RCON credentials disabled. Nine top-level pages were examined at 1440×1000 and 390×844. Populated Fleet, server details, and beta scenarios used synthetic API responses. Beta setup, blueprint validation, and the browser accessibility tree were also exercised. Navigation was checked around its responsive breakpoint. Production integrations, actual restarts, and destructive actions were not executed. Browser exploration used an isolated database with external connections disabled. Nine top-level pages were examined at 1440×1000 and 390×844. Populated Fleet, server details, and beta scenarios used synthetic API responses. Beta setup, blueprint validation, and the browser accessibility tree were also exercised. Navigation was checked around its responsive breakpoint. Production integrations, actual restarts, and destructive actions were not executed.
**Evidence labels:** **Rendered** means reproduced in the local browser, sometimes with an explicitly synthetic populated state. **Source** means verified in UI/API implementation. **Opportunity** means a proposed capability or design direction, rather than a defect. Recommendations are expert judgments; they are not findings from end-user interviews or a complete accessibility conformance audit. **Evidence labels:** **Rendered** means reproduced in the local browser, sometimes with an explicitly synthetic populated state. **Source** means verified in UI/API implementation. **Opportunity** means a proposed capability or design direction, rather than a defect. Recommendations are expert judgments; they are not findings from end-user interviews or a complete accessibility conformance audit.
@@ -198,7 +198,7 @@ Sources: [Scripts](../src/routes/scripts/+page.svelte), lines 13–23; [Plugins]
Settings currently leads only to server keys. Integration information is available deep in server diagnostics; the empty Fleet encourages creation even when external services are not configured. The first-run user needs a short path from prerequisites to a functioning lobby. Settings currently leads only to server keys. Integration information is available deep in server diagnostics; the empty Fleet encourages creation even when external services are not configured. The first-run user needs a short path from prerequisites to a functioning lobby.
Add a read-only System status page and a contextual first-run checklist: Komodo configured/reachable, RCON configured, game files available where verifiable, keys available, and first server ready. Link directly to relevant setup instructions. Distinguish **not configured**, **unreachable**, and **server offline**. Avoid repeatedly displaying the full checklist after setup succeeds. Add a read-only System status page and a contextual first-run checklist: host agents enrolled/connected, local RCON responding, verified game files available, keys available, and first server ready. Link directly to relevant setup instructions. Distinguish **not configured**, **unreachable**, and **server offline**. Avoid repeatedly displaying the full checklist after setup succeeds.
Acceptance: someone with an empty installation can identify the next missing prerequisite without opening an unrelated server or reading the full README. Acceptance: someone with an empty installation can identify the next missing prerequisite without opening an unrelated server or reading the full README.
@@ -306,7 +306,7 @@ The visual language is coherent, but small controls, nested cards, repeated sect
Use a dedicated editable-control boundary token, reserve strong surfaces for actionable sections, and reduce repeated title/subtitle layers in builders. Keep body text and important form instructions comfortably readable; use compact metadata selectively. Add descriptive route titles, especially server name and lobby number. Use a dedicated editable-control boundary token, reserve strong surfaces for actionable sections, and reduce repeated title/subtitle layers in builders. Keep body text and important form instructions comfortably readable; use compact metadata selectively. Add descriptive route titles, especially server name and lobby number.
Use “server” consistently for the managed unit; introduce “lobby” as its player-facing meaning. Prefer **Preview missing servers** over “Conform fleet.” Use “Files” consistently for scripts/configs/plugins where a collective noun is needed. Keep raw IDs, RCON, Komodo, and hashes accessible in technical details. Expose the existing keyboard shortcuts through a small help panel. Use “server” consistently for the managed unit; introduce “lobby” as its player-facing meaning. Prefer **Preview missing servers** over “Conform fleet.” Use “Files” consistently for scripts/configs/plugins where a collective noun is needed. Keep raw IDs, RCON, host-agent status, and hashes accessible in technical details. Expose the existing keyboard shortcuts through a small help panel.
Acceptance: primary tasks are identifiable by placement and wording, and multiple browser tabs are distinguishable. Contrast should be measured before claiming a conformance result. Acceptance: primary tasks are identifiable by placement and wording, and multiple browser tabs are distinguishable. Contrast should be measured before claiming a conformance result.
-41
View File
@@ -1,41 +0,0 @@
# Komodo stack definition. Deploys electric-cherry as a managed container on
# the same host running the T5/T6 Plutonium servers. Mounts /root/zombies so the
# app can push scripts to the same paths push_scripts.sh used, and dedicates
# a small subdir for its SQLite state.
[[stack]]
name = "electric-cherry"
[stack.services.web]
image = "electric-cherry:latest"
restart = "unless-stopped"
ports = ["3000:3000"]
volumes = [
"/root/zombies:/root/zombies:rw",
"/root/zombies/.electric-cherry:/data:rw"
]
# Lets RCON_HOST=host.docker.internal reach game servers running on the
# same Komodo host without knowing each container's internal IP.
extra_hosts = ["host.docker.internal:host-gateway"]
[stack.services.web.env]
SERVERS_ROOT = "/root/zombies/t5"
T6_SERVERS_ROOT = "/root/zombies/t6"
PLUTAINER_T6_GAMEFILES_VOLUME = "t6_game_files"
REL_DEST = "runtime/plutonium/storage/t5/raw/scripts/sp"
BETA_ARCHIVE_DIR = "/root/zombies/beta-archive"
DB_PATH = "/data/cherry.db"
# secrets: bind via Komodo's env-file or secret store
KOMODO_ADDRESS = "https://komodo.example.internal"
KOMODO_API_KEY = ""
KOMODO_API_SECRET = ""
# Host new deployments land on — must match the name shown in Komodo's UI.
KOMODO_TARGET_SERVER = ""
# RCON — shared password across every Plutonium server in the fleet.
RCON_PASSWORD = ""
RCON_HOST = "host.docker.internal"
RCON_TIMEOUT_MS = "3000"
+4 -4
View File
@@ -1,12 +1,12 @@
import type { ProvisioningAction } from './provisioning.js'; import type { ProvisioningAction } from './provisioning.js';
import type { LaunchRecipeSummary } from './deployment-plan.js'; import type { LaunchRecipeSummary } from './deployment-plan.js';
export const provisioningPhases = ['requested', 'eligibility', 'startup-queue', 'prepare', 'create-deployment', export const provisioningPhases = ['requested', 'eligibility', 'startup-queue', 'prepare',
'container-action', 'remove-container', 'delete-deployment', 'cleanup', 'verify-container', 'verify-game', 'container-action', 'cleanup', 'verify-container', 'verify-game',
'verify-map', 'verify-plutonium', 'verify-stopped'] as const; 'verify-map', 'verify-plutonium', 'verify-stopped'] as const;
export type ProvisioningPhase = typeof provisioningPhases[number]; export type ProvisioningPhase = typeof provisioningPhases[number];
export type EvidenceResult = 'pass' | 'waiting' | 'fail' | 'unknown'; export type EvidenceResult = 'pass' | 'waiting' | 'fail' | 'unknown';
export type EvidenceSource = 'komodo' | 'agent' | 'rcon' | 'telemetry' | 'combined' | 'manifest' | null; export type EvidenceSource = 'agent' | 'rcon' | 'telemetry' | 'combined' | 'manifest' | null;
export interface ProvisioningCheck { export interface ProvisioningCheck {
key: 'container' | 'game' | 'map' | 'plutonium-release' | 'plutonium-version' | 'occupancy' | 'stopped'; key: 'container' | 'game' | 'map' | 'plutonium-release' | 'plutonium-version' | 'occupancy' | 'stopped';
@@ -98,7 +98,7 @@ export interface ProvisioningDetail {
canCancel?: boolean; canCancel?: boolean;
canCancelCreation?: boolean; canCancelCreation?: boolean;
cancellation?: { cleanupId: string; state: string; stage: string; error: string | null } | null; cancellation?: { cleanupId: string; state: string; stage: string; error: string | null } | null;
backend?: 'native' | 'komodo'; backend?: 'native';
workerActivity?: WorkerActivity[]; workerActivity?: WorkerActivity[];
agentOperation?: { id: string; hostId: string; state: string; updatedAt: string; createdAt: string; error: string | null } | null; agentOperation?: { id: string; hostId: string; state: string; updatedAt: string; createdAt: string; error: string | null } | null;
deployment: ProvisioningDeployment; deployment: ProvisioningDeployment;
+1 -29
View File
@@ -33,31 +33,7 @@ export const config = {
betaArchiveDir: req('BETA_ARCHIVE_DIR', '/root/zombies/beta-archive'), betaArchiveDir: req('BETA_ARCHIVE_DIR', '/root/zombies/beta-archive'),
dbPath: req('DB_PATH', './data/cherry.db'), dbPath: req('DB_PATH', './data/cherry.db'),
httpTimeoutMs: num('HTTP_TIMEOUT_MS', 10_000), httpTimeoutMs: num('HTTP_TIMEOUT_MS', 10_000),
komodo: { defaultRuntimeImage: env.PLUTAINER_IMAGE ?? 'ghcr.io/ayymoss/plutainer:latest',
address: env.KOMODO_ADDRESS ?? '',
apiKey: env.KOMODO_API_KEY ?? '',
apiSecret: env.KOMODO_API_SECRET ?? '',
// Komodo server (host) new deployments land on. Required for the
// deploy flow — Komodo needs to know which machine to schedule the
// container onto. Look up the exact name in Komodo's UI (Servers page).
targetServer: env.KOMODO_TARGET_SERVER ?? '',
// Default image for freshly-provisioned T5 and T6 zombie servers.
// Overridable per-deploy in the UI later if we ever need to.
defaultImage: env.PLUTAINER_IMAGE ?? 'ghcr.io/ayymoss/plutainer:latest',
defaultNetwork: env.PLUTAINER_NETWORK ?? 'komodo_default',
// Named docker volume that holds the read-only game files layer.
gameFilesVolume: env.PLUTAINER_GAMEFILES_VOLUME ?? 't5_game_files',
t6GameFilesVolume: env.PLUTAINER_T6_GAMEFILES_VOLUME ?? 't6_game_files'
},
// Shared across every newly-provisioned server. Kept in sync with the
// fleet's RCON_PASSWORD (rcon.password) so freshly-created servers respond
// to the same rcon secret the sweep already uses.
// Port range Electric Cherry auto-allocates from when creating a new
// deployment. Update this when the host's firewall opens more.
portRange: {
min: num('SERVER_PORT_MIN', 28960),
max: num('SERVER_PORT_MAX', 28980)
},
rcon: { rcon: {
// Host every RCON packet is sent to. Defaults to the Docker // Host every RCON packet is sent to. Defaults to the Docker
// host-gateway alias so the electric-cherry container can reach // host-gateway alias so the electric-cherry container can reach
@@ -70,10 +46,6 @@ export const config = {
} }
} as const; } as const;
export function komodoConfigured(): boolean {
return !!(config.komodo.address && config.komodo.apiKey && config.komodo.apiSecret);
}
export function rconConfigured(): boolean { export function rconConfigured(): boolean {
return !!config.rcon.password; return !!config.rcon.password;
} }
+9 -2
View File
@@ -24,7 +24,6 @@ const SERVERS_COLUMNS: Array<{ name: string; ddl: string }> = [
{ name: 'desired_state', ddl: "ALTER TABLE servers ADD COLUMN desired_state TEXT NOT NULL DEFAULT 'running' CHECK (desired_state IN ('running','stopped'))" }, { name: 'desired_state', ddl: "ALTER TABLE servers ADD COLUMN desired_state TEXT NOT NULL DEFAULT 'running' CHECK (desired_state IN ('running','stopped'))" },
// Base columns that shipped in the very first schema stay here for parity // Base columns that shipped in the very first schema stay here for parity
// with fresh installs — no ALTER is issued if they're already present. // with fresh installs — no ALTER is issued if they're already present.
{ name: 'komodo_server', ddl: 'ALTER TABLE servers ADD COLUMN komodo_server TEXT' },
{ name: 'game_port', ddl: 'ALTER TABLE servers ADD COLUMN game_port INTEGER' }, { name: 'game_port', ddl: 'ALTER TABLE servers ADD COLUMN game_port INTEGER' },
{ name: 'rcon_host', ddl: 'ALTER TABLE servers ADD COLUMN rcon_host TEXT' }, { name: 'rcon_host', ddl: 'ALTER TABLE servers ADD COLUMN rcon_host TEXT' },
{ name: 'last_players', ddl: 'ALTER TABLE servers ADD COLUMN last_players INTEGER' }, { name: 'last_players', ddl: 'ALTER TABLE servers ADD COLUMN last_players INTEGER' },
@@ -65,6 +64,14 @@ function migrateServers(handle: Database.Database): void {
handle.exec(col.ddl); handle.exec(col.ddl);
} }
} }
// Retire external-backend registry fields, including required fields that
// would otherwise prevent native inserts into an upgraded database.
const retired = new Set(['komodo_type', 'komodo_name', 'komodo_server']);
handle.transaction(() => {
for (const column of rows) {
if (retired.has(column.name)) handle.exec(`ALTER TABLE servers DROP COLUMN "${column.name.replaceAll('"', '""')}"`);
}
})();
} }
// Additive columns on tables the script model owns. Same pattern as // Additive columns on tables the script model owns. Same pattern as
@@ -286,7 +293,7 @@ function migrateWatchdogConstraints(handle: Database.Database): void {
function backfillDesiredState(handle: Database.Database): void { function backfillDesiredState(handle: Database.Database): void {
if (handle.prepare("SELECT 1 FROM settings WHERE key = 'watchdog_desired_state_v1'").get()) return; if (handle.prepare("SELECT 1 FROM settings WHERE key = 'watchdog_desired_state_v1'").get()) return;
handle.transaction(() => { handle.transaction(() => {
// A failed stop may have reached Komodo. Preserve that explicit intent. // A failed stop may have reached the host agent. Preserve that explicit intent.
handle.exec(`UPDATE servers SET desired_state = 'stopped' WHERE 'stop' = ( handle.exec(`UPDATE servers SET desired_state = 'stopped' WHERE 'stop' = (
SELECT action FROM server_deployments d WHERE d.server = servers.name AND d.state != 'cancelled' SELECT action FROM server_deployments d WHERE d.server = servers.name AND d.state != 'cancelled'
AND d.action IN ('create','start','restart','rebuild','stop') AND d.action IN ('create','start','restart','rebuild','stop')
+2 -5
View File
@@ -63,13 +63,10 @@ CREATE INDEX IF NOT EXISTS idx_leaderboard_player_matches ON leaderboard_match_p
CREATE TABLE IF NOT EXISTS servers ( CREATE TABLE IF NOT EXISTS servers (
name TEXT PRIMARY KEY, name TEXT PRIMARY KEY,
komodo_type TEXT NOT NULL DEFAULT 'deployment', -- deployment|stack|container
komodo_name TEXT NOT NULL,
komodo_server TEXT,
container_checked_at TEXT, container_checked_at TEXT,
container_state TEXT, -- last observed Komodo runtime state container_state TEXT, -- last observed host-agent runtime state
desired_state TEXT NOT NULL DEFAULT 'running' CHECK (desired_state IN ('running','stopped')), desired_state TEXT NOT NULL DEFAULT 'running' CHECK (desired_state IN ('running','stopped')),
game_port INTEGER, -- UDP RCON port; lazy-filled from Komodo game_port INTEGER, -- UDP game port assigned on the host
rcon_host TEXT, -- optional per-server override of RCON_HOST rcon_host TEXT, -- optional per-server override of RCON_HOST
game TEXT, -- 't5' etc.; static per-server (no map rotation) game TEXT, -- 't5' etc.; static per-server (no map rotation)
map TEXT, -- 'zombie_theater' etc.; static per-server map TEXT, -- 'zombie_theater' etc.; static per-server
+1 -1
View File
@@ -376,7 +376,7 @@ export async function openBeta(input: OpenBetaInput): Promise<OpenBetaResult> {
const removal = db().prepare(`SELECT id FROM server_deployments WHERE server = ? AND server_deleted_at IS NULL const removal = db().prepare(`SELECT id FROM server_deployments WHERE server = ? AND server_deleted_at IS NULL
AND action IN ('destroy', 'rebuild') AND (state IN ('waiting', 'queued', 'running', 'awaiting-health') AND action IN ('destroy', 'rebuild') AND (state IN ('waiting', 'queued', 'running', 'awaiting-health')
OR (state IN ('failed', 'interrupted') AND stage IN ( OR (state IN ('failed', 'interrupted') AND stage IN (
'Stopping and removing container', 'Deleting Komodo deployment', 'Removing server files and registry'))) 'Stopping and removing container', 'Removing server files and registry')))
LIMIT 1`).get(name); LIMIT 1`).get(name);
if (removal) throw new BetaError(`${name}: cancel or finish the pending destruction or rebuild before opening a beta.`, 'conflict'); if (removal) throw new BetaError(`${name}: cancel or finish the pending destruction or rebuild before opening a beta.`, 'conflict');
serverRows.push({ name, game: row.game, map: row.map }); serverRows.push({ name, game: row.game, map: row.map });
+18 -47
View File
@@ -2,16 +2,14 @@ import { fleetSettings } from '../settings.js';
import { randomUUID } from 'node:crypto'; import { randomUUID } from 'node:crypto';
import type { CollectionDetail, CollectionMode, CollectionPreview, CollectionSummary } from '$lib/deployment-plan.js'; import type { CollectionDetail, CollectionMode, CollectionPreview, CollectionSummary } from '$lib/deployment-plan.js';
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import { config, komodoConfigured, rconConfigured } from '../config.js';
import { blueprintEntries, summarizeRecipe, type BlueprintEntry } from './launch-recipes.js'; import { blueprintEntries, summarizeRecipe, type BlueprintEntry } from './launch-recipes.js';
import { creationQueue, destroyServer, nextAvailablePort, queueCreateServer } from '../provisioning/index.js'; import { creationQueue, destroyServer } from '../provisioning/index.js';
import { deploymentStage, finishDeployment, recordDeploymentEvent } from './deployments.js'; import { deploymentStage, finishDeployment, recordDeploymentEvent } from './deployments.js';
import { housePartyTargets } from './house-party.js'; import { housePartyTargets } from './house-party.js';
import { serverObservation } from './status.js'; import { serverObservation } from './status.js';
import { activeBetaRemovalBlocker, getServer } from './servers.js'; import { activeBetaRemovalBlocker, getServer } from './servers.js';
import { parseServerName, zombieMapBySlug, zombieMaps } from './map-aliases.js'; import { parseServerName, zombieMapBySlug, zombieMaps } from './map-aliases.js';
import { nativeLaunchSelection, queueNativeCreationIntent } from '../provisioning/native.js'; import { nativeLaunchSelection, queueNativeCreationIntent } from '../provisioning/native.js';
import { listHosts } from '../hosts/index.js';
import { nativeDeployment, nativeInstance } from '../backends/native.js'; import { nativeDeployment, nativeInstance } from '../backends/native.js';
import { launchAvailability } from '../hosts/availability.js'; import { launchAvailability } from '../hosts/availability.js';
@@ -71,7 +69,7 @@ function collectionPlan(entries: BlueprintEntry[], mode: CollectionMode, include
return { counts, retainedServers, removals }; return { counts, retainedServers, removals };
} }
function allocate(entries: BlueprintEntry[], releasingServers: string[] = [], counts = coverage(entries)): CollectionPreview['servers'] { function allocate(entries: BlueprintEntry[], counts = coverage(entries)): CollectionPreview['servers'] {
const occupied = db().prepare(`SELECT name FROM servers UNION const occupied = db().prepare(`SELECT name FROM servers UNION
SELECT j.server AS name FROM server_creation_jobs j JOIN server_deployments d ON d.id=j.deployment_id SELECT j.server AS name FROM server_creation_jobs j JOIN server_deployments d ON d.id=j.deployment_id
WHERE d.server_deleted_at IS NULL AND (d.state IN ('queued','running','awaiting-health') WHERE d.server_deleted_at IS NULL AND (d.state IN ('queued','running','awaiting-health')
@@ -85,16 +83,14 @@ function allocate(entries: BlueprintEntry[], releasingServers: string[] = [], co
taken.add(Number(match[2])); lobbies.set(match[1], taken); taken.add(Number(match[2])); lobbies.set(match[1], taken);
} }
const savedKeys = new Set((db().prepare("SELECT server_id FROM server_keys WHERE length(trim(pluto_key))>0").all() as { server_id: string }[]).map(row => row.server_id)); const savedKeys = new Set((db().prepare("SELECT server_id FROM server_keys WHERE length(trim(pluto_key))>0").all() as { server_id: string }[]).map(row => row.server_id));
const ports: number[] = [];
return entries.flatMap((entry, entryIndex) => Array.from({ length: counts[entryIndex].missing }, () => { return entries.flatMap((entry, entryIndex) => Array.from({ length: counts[entryIndex].missing }, () => {
const base = `${entry.recipe.game}_zombie_${entry.recipe.mapSlug}`; const base = `${entry.recipe.game}_zombie_${entry.recipe.mapSlug}`;
const taken = lobbies.get(base) ?? new Set<number>(); const taken = lobbies.get(base) ?? new Set<number>();
const lobby = Array.from({ length: 20 }, (_, index) => index + 1).find(number => !taken.has(number)) ?? null; const lobby = Array.from({ length: 20 }, (_, index) => index + 1).find(number => !taken.has(number)) ?? null;
const name = lobby === null ? null : `${base}_${lobby}`; const name = lobby === null ? null : `${base}_${lobby}`;
const native = entry.recipe.native; const native = entry.recipe.native;
const port = native ? null : nextAvailablePort(ports, releasingServers); const port = null;
if (lobby !== null) { taken.add(lobby); lobbies.set(base, taken); } if (lobby !== null) { taken.add(lobby); lobbies.set(base, taken); }
if (port !== null) ports.push(port);
return { entryIndex, name, port, ...(native ? { native } : {}), hasSavedKey: !!name && savedKeys.has(name) }; return { entryIndex, name, port, ...(native ? { native } : {}), hasSavedKey: !!name && savedKeys.has(name) };
})); }));
} }
@@ -102,13 +98,8 @@ function allocate(entries: BlueprintEntry[], releasingServers: string[] = [], co
function blockersFor(servers: CollectionPreview['servers'], entries: BlueprintEntry[] = []): string[] { function blockersFor(servers: CollectionPreview['servers'], entries: BlueprintEntry[] = []): string[] {
const blockers: string[] = []; const blockers: string[] = [];
if (!servers.length) return blockers; if (!servers.length) return blockers;
const legacy = servers.filter(server => !server.native && !entries[server.entryIndex]?.recipe.native); if (servers.some(server => !server.native && !entries[server.entryIndex]?.recipe.native)) blockers.push('Select and prepare a host before launching.');
if (legacy.length) {
if (!komodoConfigured() || !config.komodo.targetServer) blockers.push('Select and prepare a host before launching.');
if (!rconConfigured()) blockers.push('Configure RCON_PASSWORD so every new server can be verified.');
}
if (servers.some(server => !server.name)) blockers.push('There are not enough free lobbies for one or more maps (20 per map). Reduce their counts or free capacity.'); if (servers.some(server => !server.name)) blockers.push('There are not enough free lobbies for one or more maps (20 per map). Reduce their counts or free capacity.');
if (legacy.some(server => server.port === null)) blockers.push(`There are not enough free game ports in ${config.portRange.min}–${config.portRange.max} for the whole collection.`);
return blockers; return blockers;
} }
@@ -120,7 +111,7 @@ function removalBlockersFor(removals: CollectionPreview['removals']): string[] {
AND action='create' AND state IN ('queued','running','awaiting-health') ORDER BY rowid DESC LIMIT 1`).get(name) as { id: string } | undefined; AND action='create' AND state IN ('queued','running','awaiting-health') ORDER BY rowid DESC LIMIT 1`).get(name) as { id: string } | undefined;
return !!creation && nativeDeployment(creation.id); return !!creation && nativeDeployment(creation.id);
}; };
if (removals.some(server => !native(server.name)) && (!komodoConfigured() || !rconConfigured())) blockers.push('Configure the legacy backend before removing its servers.'); if (removals.some(server => !native(server.name))) blockers.push('Every server to remove must have a native host instance.');
for (const server of removals) { for (const server of removals) {
const blocker = activeBetaRemovalBlocker(server.name); const blocker = activeBetaRemovalBlocker(server.name);
if (blocker) blockers.push(blocker); if (blocker) blockers.push(blocker);
@@ -133,15 +124,13 @@ export function previewCollection(blueprintId: unknown, mode: unknown = 'fill',
if (mode !== 'fill' && mode !== 'exact') throw new Error('Choose fill or exact collection mode.'); if (mode !== 'fill' && mode !== 'exact') throw new Error('Choose fill or exact collection mode.');
return db().transaction((): CollectionPreview => { return db().transaction((): CollectionPreview => {
const { entries, blueprintName } = blueprintEntries(blueprintId); const { entries, blueprintName } = blueprintEntries(blueprintId);
if (selection?.hostId !== undefined || !config.komodo.address || listHosts().length > 0) { const libraries = selection?.libraryIds as Record<string, unknown> | undefined;
const libraries = selection?.libraryIds as Record<string, unknown> | undefined; for (const entry of entries) {
for (const entry of entries) { entry.recipe.native = nativeLaunchSelection({ ...selection, libraryId: libraries?.[entry.recipe.game] ?? selection?.libraryId }, entry.recipe.game);
entry.recipe.native = nativeLaunchSelection({ ...selection, libraryId: libraries?.[entry.recipe.game] ?? selection?.libraryId }, entry.recipe.game); entry.recipe.image = entry.recipe.native.image!;
entry.recipe.image = entry.recipe.native.image!;
}
} }
const { counts, retainedServers, removals } = collectionPlan(entries, mode); const { counts, retainedServers, removals } = collectionPlan(entries, mode);
const servers = allocate(entries, removals.map(server => server.name), counts); const servers = allocate(entries, counts);
const blockers = [...blockersFor(servers, entries), ...removalBlockersFor(removals)]; const blockers = [...blockersFor(servers, entries), ...removalBlockersFor(removals)];
const selections = servers.flatMap(server => server.native ? [server.native] : []); const selections = servers.flatMap(server => server.native ? [server.native] : []);
const availability = selections.length ? launchAvailability(selections) : {}; const availability = selections.length ? launchAvailability(selections) : {};
@@ -176,16 +165,12 @@ export function queueCollection(input: { previewId?: unknown; keys?: unknown }):
throw new Error('The fleet changed. Nothing was queued. Refresh the preview and review the servers to keep and remove.'); throw new Error('The fleet changed. Nothing was queued. Refresh the preview and review the servers to keep and remove.');
} }
const { counts, retainedServers, removals } = collectionPlan(entries, preview.mode, false); const { counts, retainedServers, removals } = collectionPlan(entries, preview.mode, false);
const releasingServers = removals.map(server => server.name); const servers = reviewed;
const nativeReview = reviewed.every(server => server.native || entries[server.entryIndex]?.recipe.native);
// Native Apply accepts the reviewed logical identities. Capacity and
// physical ports can change while this request waits in the queue.
const servers = nativeReview ? reviewed : allocate(entries, releasingServers, counts);
const validatedSelections = new Set<string>(); const validatedSelections = new Set<string>();
for (const server of servers) { for (const server of servers) {
const recipe = entries[server.entryIndex].recipe; const recipe = entries[server.entryIndex].recipe;
const selection = server.native ?? recipe.native; const selection = server.native ?? recipe.native;
if (!selection) continue; if (!selection) throw new Error('Select a native host before launching. Refresh the preview.');
const identity = JSON.stringify([recipe.game, selection]); const identity = JSON.stringify([recipe.game, selection]);
if (validatedSelections.has(identity)) continue; if (validatedSelections.has(identity)) continue;
// Recheck saved configuration once per selection without allocating // Recheck saved configuration once per selection without allocating
@@ -195,9 +180,8 @@ export function queueCollection(input: { previewId?: unknown; keys?: unknown }):
} }
const blockers = [...blockersFor(servers, entries), ...removalBlockersFor(removals)]; const blockers = [...blockersFor(servers, entries), ...removalBlockersFor(removals)];
if (blockers.length) throw new Error(blockers.join(' ')); if (blockers.length) throw new Error(blockers.join(' '));
if (entries.some((_entry, index) => counts[index].missing !== reviewed.filter(server => server.entryIndex === index).length) if (entries.some((_entry, index) => counts[index].missing !== reviewed.filter(server => server.entryIndex === index).length)) {
|| servers.length !== reviewed.length || servers.some((server, i) => server.entryIndex !== reviewed[i]?.entryIndex || server.name !== reviewed[i]?.name || (!server.native && server.port !== reviewed[i]?.port))) { throw new Error('Fleet counts changed. Nothing was queued. Refresh the preview and review each server key.');
throw new Error('Fleet counts, lobby or port availability changed. Nothing was queued. Refresh the preview and review each server key.');
} }
const expected = new Set(servers.map(server => server.name)); const expected = new Set(servers.map(server => server.name));
for (const [name, value] of Object.entries(keys)) { for (const [name, value] of Object.entries(keys)) {
@@ -207,10 +191,7 @@ export function queueCollection(input: { previewId?: unknown; keys?: unknown }):
const collectionId = randomUUID(); const collectionId = randomUUID();
db().prepare('INSERT INTO server_launch_collections (id, name, created_at, mode, retained_servers) VALUES (?, ?, ?, ?, ?)') db().prepare('INSERT INTO server_launch_collections (id, name, created_at, mode, retained_servers) VALUES (?, ?, ?, ?, ?)')
.run(collectionId, preview.blueprint_name, new Date().toISOString(), preview.mode, JSON.stringify(retainedServers)); .run(collectionId, preview.blueprint_name, new Date().toISOString(), preview.mode, JSON.stringify(retainedServers));
const removalJobs = removals.map((server, position) => { removals.forEach((server, position) => {
const ports = (db().prepare(`SELECT game_port AS port FROM servers WHERE name = ? UNION
SELECT j.port FROM server_creation_jobs j JOIN server_deployments d ON d.id = j.deployment_id
WHERE j.server = ? AND d.server_deleted_at IS NULL AND d.state IN ('queued', 'running', 'awaiting-health')`).all(server.name, server.name) as { port: number | null }[]).map(row => row.port);
const queued = db().prepare("SELECT id FROM server_deployments WHERE server = ? AND server_deleted_at IS NULL AND action = 'create' AND state = 'queued'").get(server.name) as { id: string } | undefined; const queued = db().prepare("SELECT id FROM server_deployments WHERE server = ? AND server_deleted_at IS NULL AND action = 'create' AND state = 'queued'").get(server.name) as { id: string } | undefined;
let id: string; let id: string;
if (queued && !getServer(server.name)) { if (queued && !getServer(server.name)) {
@@ -230,27 +211,17 @@ export function queueCollection(input: { previewId?: unknown; keys?: unknown }):
id = result.deploymentId; id = result.deploymentId;
} }
db().prepare('INSERT INTO server_collection_removals (collection_id, deployment_id, position) VALUES (?, ?, ?)').run(collectionId, id, position); db().prepare('INSERT INTO server_collection_removals (collection_id, deployment_id, position) VALUES (?, ?, ?)').run(collectionId, id, position);
return { id, ports };
}); });
servers.forEach((server, index) => { servers.forEach((server, index) => {
const recipe = { ...entries[server.entryIndex].recipe, ...(server.native ? { native: server.native } : {}) }; const recipe = { ...entries[server.entryIndex].recipe, ...(server.native ? { native: server.native } : {}) };
let result; if (!recipe.native) throw new Error('Select a native host before launching. Refresh the preview.');
if (recipe.native) result = queueNativeCreationIntent({ server: server.name!, game: recipe.game, mapSlug: recipe.mapSlug, const result = queueNativeCreationIntent({ server: server.name!, game: recipe.game, mapSlug: recipe.mapSlug,
selection: recipe.native, recipe, plutoKey: keys[server.name!] as string | undefined, createdBy: 'ui', blueprintName: preview.blueprint_name }); selection: recipe.native, recipe, plutoKey: keys[server.name!] as string | undefined, createdBy: 'ui', blueprintName: preview.blueprint_name });
else {
const childPreviewId = randomUUID();
db().prepare('INSERT INTO server_launch_previews (id, expires_at, server, port, recipe, blueprint_name) VALUES (?, ?, ?, ?, ?, ?)')
.run(childPreviewId, preview.expires_at, server.name, server.port, JSON.stringify(recipe), preview.blueprint_name);
result = queueCreateServer({ game: recipe.game, gameMode: 'zombie', mapSlug: recipe.mapSlug,
expectedName: server.name!, previewId: childPreviewId, plutoKey: keys[server.name!] as string | undefined, createdBy: 'ui' }, releasingServers);
}
// Throw so the outer transaction also reverses all earlier keys and children. // Throw so the outer transaction also reverses all earlier keys and children.
if (!result.ok) throw new Error(`${server.name}: ${result.message} Nothing was queued.`); if (!result.ok) throw new Error(`${server.name}: ${result.message} Nothing was queued.`);
db().prepare('INSERT INTO server_collection_members (collection_id, deployment_id, position) VALUES (?, ?, ?)') db().prepare('INSERT INTO server_collection_members (collection_id, deployment_id, position) VALUES (?, ?, ?)')
.run(collectionId, result.deploymentId, index); .run(collectionId, result.deploymentId, index);
for (const removal of recipe.native ? [] : removalJobs.filter(removal => removal.ports.includes(server.port))) {
db().prepare('INSERT INTO server_creation_dependencies (deployment_id, removal_id) VALUES (?, ?)').run(result.deploymentId, removal.id);
}
}); });
db().prepare('UPDATE server_collection_previews SET collection_id = ? WHERE id = ?').run(collectionId, preview.id); db().prepare('UPDATE server_collection_previews SET collection_id = ? WHERE id = ?').run(collectionId, preview.id);
return { ok: true as const, collectionId }; return { ok: true as const, collectionId };
+6 -13
View File
@@ -3,7 +3,7 @@ import { config } from '../config.js';
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import { getHost, queueHostOperation, waitHostOperationStatus } from '../hosts/index.js'; import { getHost, queueHostOperation, waitHostOperationStatus } from '../hosts/index.js';
import { nativeInstance, type NativeInstance } from '../backends/native.js'; import { nativeInstance, type NativeInstance } from '../backends/native.js';
import { komodoInspectHealthy } from '../integrations/komodo.js'; import { nativeHealth } from '../backends/native.js';
import { rconStatusResult, agentRconStatusResult, type AgentRconReply } from '../integrations/rcon.js'; import { rconStatusResult, agentRconStatusResult, type AgentRconReply } from '../integrations/rcon.js';
import { withOperationSource } from '../hosts/operation-context.js'; import { withOperationSource } from '../hosts/operation-context.js';
import { getServer } from './servers.js'; import { getServer } from './servers.js';
@@ -39,7 +39,6 @@ function identity(server: string, probe: Probe): Identity {
return { return {
key: JSON.stringify([row.created_at, row.desired_state, latestGameSubmission(server), lifecycle, key: JSON.stringify([row.created_at, row.desired_state, latestGameSubmission(server), lifecycle,
native ? [native.host_id, native.id, native.generation] : null, native ? [native.host_id, native.id, native.generation] : null,
native ? null : [row.komodo_type, row.komodo_name, row.komodo_server, config.komodo, config.httpTimeoutMs],
probe === 'rcon' ? [row.rcon_host, config.rcon] : null]), probe === 'rcon' ? [row.rcon_host, config.rcon] : null]),
port: probe === 'rcon' ? native?.port ?? row.game_port : null, port: probe === 'rcon' ? native?.port ?? row.game_port : null,
checkedAt: probe === 'rcon' ? row.last_checked_at : row.container_checked_at checkedAt: probe === 'rcon' ? row.last_checked_at : row.container_checked_at
@@ -77,13 +76,7 @@ async function observe<T extends { ok: boolean }>(server: string, fresh: boolean
while (entries.size > MAX_SERVERS) entries.delete(entries.keys().next().value!); while (entries.size > MAX_SERVERS) entries.delete(entries.keys().next().value!);
const pending = Promise.resolve().then(() => request(server)).then(result => { const pending = Promise.resolve().then(() => request(server)).then(result => {
const latest = identity(server, probe); const latest = identity(server, probe);
// Legacy RCON may discover and save its first port during this request. if (!sameIdentity(current, latest) || newerEvidence(latest, entry.startedAt)) {
// Accept that enrichment only when all other identity fields agree.
const endpoint = 'endpoint' in result ? result.endpoint : null;
const learnedPort = probe === 'rcon' && !current.port && result.ok
&& endpoint !== null && typeof endpoint === 'object' && 'port' in endpoint && endpoint.port === latest.port;
if (current.key !== latest.key || (current.port !== latest.port && !learnedPort)
|| newerEvidence(latest, entry.startedAt)) {
throw new MonitoringChangedError('Server changed during monitoring; the previous observation was discarded.'); throw new MonitoringChangedError('Server changed during monitoring; the previous observation was discarded.');
} }
const observation = { result, observedAt: new Date().toISOString() }; const observation = { result, observedAt: new Date().toISOString() };
@@ -100,12 +93,12 @@ async function observe<T extends { ok: boolean }>(server: string, fresh: boolean
return pending; return pending;
} }
const inspections = new Map<string, Entry<Awaited<ReturnType<typeof komodoInspectHealthy>>>>(); const inspections = new Map<string, Entry<Awaited<ReturnType<typeof nativeHealth>>>>();
const rcon = new Map<string, Entry<Awaited<ReturnType<typeof rconStatusResult>>>>(); const rcon = new Map<string, Entry<Awaited<ReturnType<typeof rconStatusResult>>>>();
interface MonitoringStatus { interface MonitoringStatus {
identity: string; identity: string;
container: Observation<Awaited<ReturnType<typeof komodoInspectHealthy>>>; container: Observation<Awaited<ReturnType<typeof nativeHealth>>>;
game?: Observation<Awaited<ReturnType<typeof rconStatusResult>>>; game?: Observation<Awaited<ReturnType<typeof rconStatusResult>>>;
} }
// Keep failed samples too, independently of the successful-probe cache. Reading // Keep failed samples too, independently of the successful-probe cache. Reading
@@ -160,7 +153,7 @@ export function monitoringStatus(server: string): MonitoringStatus | null {
} }
/** Keep the actual sample time: a later heartbeat must not renew old evidence. */ /** Keep the actual sample time: a later heartbeat must not renew old evidence. */
export function heartbeatInspection(server: string): Observation<Awaited<ReturnType<typeof komodoInspectHealthy>>> | null { export function heartbeatInspection(server: string): Observation<Awaited<ReturnType<typeof nativeHealth>>> | null {
const row = getServer(server); const row = getServer(server);
if (!row?.native_instance_id || !row.host_id) return null; if (!row?.native_instance_id || !row.host_id) return null;
const host = getHost(row.host_id); const host = getHost(row.host_id);
@@ -242,7 +235,7 @@ export function inspectForMonitoring(server: string, fresh = false) {
if (fresh) statuses.delete(server); if (fresh) statuses.delete(server);
const heartbeat = !fresh && heartbeatInspection(server); const heartbeat = !fresh && heartbeatInspection(server);
if (heartbeat) return Promise.resolve(heartbeat); if (heartbeat) return Promise.resolve(heartbeat);
return observe(server, fresh, 'inspect', inspections, komodoInspectHealthy); return observe(server, fresh, 'inspect', inspections, nativeHealth);
} }
export function rconForMonitoring(server: string, fresh = false) { export function rconForMonitoring(server: string, fresh = false) {
if (fresh) statuses.delete(server); if (fresh) statuses.delete(server);
+1 -1
View File
@@ -105,7 +105,7 @@ function buildRecipe(game: ZombieGame, mapSlug: string, includeContent: boolean)
} }
addConfig('Selected map', MAP_CONFIG_FILE, getServerMapConfig(game, map.canonical, map.slug)!); addConfig('Selected map', MAP_CONFIG_FILE, getServerMapConfig(game, map.canonical, map.slug)!);
return { format: 1, game, mapSlug, mapName: map.friendly, expectedMap: map.canonical, return { format: 1, game, mapSlug, mapName: map.friendly, expectedMap: map.canonical,
capturedAt: new Date().toISOString(), image: config.komodo.defaultImage, capturedAt: new Date().toISOString(), image: config.defaultRuntimeImage,
configFile: configFileForGame(game), configSource, files }; configFile: configFileForGame(game), configSource, files };
} }
+3 -11
View File
@@ -74,9 +74,6 @@ export interface ServerRow {
instance_generation?: number | null; instance_generation?: number | null;
game_library_id?: string | null; game_library_id?: string | null;
name: string; name: string;
komodo_type: 'deployment' | 'stack' | 'container';
komodo_name: string;
komodo_server: string | null;
container_state: string | null; container_state: string | null;
container_checked_at: string | null; container_checked_at: string | null;
desired_state: 'running' | 'stopped'; desired_state: 'running' | 'stopped';
@@ -112,12 +109,10 @@ export interface ServerRow {
export async function seedServerRegistry(): Promise<number> { export async function seedServerRegistry(): Promise<number> {
// Explicit legacy import only; never run as part of application startup. // Explicit legacy import only; never run as part of application startup.
const rows = await serverDirectories(); const rows = await serverDirectories();
// Under the new naming scheme Komodo entities match EC server names 1:1
// (both underscore-separated), so komodo_name = name — no transformation.
const insert = db().prepare( const insert = db().prepare(
`INSERT OR IGNORE INTO servers `INSERT OR IGNORE INTO servers
(name, komodo_type, komodo_name, game, map, game_mode, map_slug, host_dir) (name, game, map, game_mode, map_slug, host_dir)
VALUES (?, 'deployment', ?, ?, ?, ?, ?, ?)` VALUES (?, ?, ?, ?, ?, ?)`
); );
// Backfill existing rows where derived columns were never set. game_mode // Backfill existing rows where derived columns were never set. game_mode
// and map_slug are new; older rows can still surface them from the name. // and map_slug are new; older rows can still surface them from the name.
@@ -139,7 +134,7 @@ export async function seedServerRegistry(): Promise<number> {
const gameMode = parsed?.gameMode ?? null; const gameMode = parsed?.gameMode ?? null;
const mapSlug = parsed?.mapSlug ?? null; const mapSlug = parsed?.mapSlug ?? null;
const hostDir = serverDir(name); const hostDir = serverDir(name);
const res = insert.run(name, name, game, map, gameMode, mapSlug, hostDir); const res = insert.run(name, game, map, gameMode, mapSlug, hostDir);
if (res.changes > 0) { if (res.changes > 0) {
n++; n++;
} else { } else {
@@ -185,9 +180,6 @@ export function setServerField(
export function setServerFields(name: string, fields: Partial<Record<keyof ServerRow, string | number | null>>): void { export function setServerFields(name: string, fields: Partial<Record<keyof ServerRow, string | number | null>>): void {
const allowed: Array<keyof ServerRow> = [ const allowed: Array<keyof ServerRow> = [
'komodo_type',
'komodo_name',
'komodo_server',
'container_state', 'container_state',
'container_checked_at', 'container_checked_at',
'game_port', 'game_port',
+3 -5
View File
@@ -1,7 +1,7 @@
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import { getServer, listServers } from './servers.js'; import { getServer, listServers } from './servers.js';
import { observationReader, serverObservation } from './status.js'; import { observationReader, serverObservation } from './status.js';
import { config, komodoConfigured, rconConfigured } from '../config.js'; import { config, rconConfigured } from '../config.js';
import { runningVersions } from '../integrations/plutonium.js'; import { runningVersions } from '../integrations/plutonium.js';
import { listHosts } from '../hosts/index.js'; import { listHosts } from '../hosts/index.js';
import type { NativeInstance } from '../backends/native.js'; import type { NativeInstance } from '../backends/native.js';
@@ -33,8 +33,7 @@ export function fleetSnapshot(serverName?: string) {
host: s.host_id ? hostsById.get(s.host_id) ?? null : null, host: s.host_id ? hostsById.get(s.host_id) ?? null : null,
network: instance ? { bindAddress: instance.bind_address, localPort: instance.port, publicAddress: instance.public_address, publicPort: instance.public_port } : null, network: instance ? { bindAddress: instance.bind_address, localPort: instance.port, publicAddress: instance.public_address, publicPort: instance.public_port } : null,
plutonium: versions.get(s.name), plutonium: versions.get(s.name),
// Derived endpoint — same logic as endpointFor() but without asking // The host agent owns the game connection.
// Komodo. If we don't have a port yet, endpoint is null.
endpoint: instance ? { host: `agent:${instance.host_id}`, port: instance.port } : s.game_port ? { host: s.rcon_host ?? rconHost, port: s.game_port } : null, endpoint: instance ? { host: `agent:${instance.host_id}`, port: instance.port } : s.game_port ? { host: s.rcon_host ?? rconHost, port: s.game_port } : null,
held: holds.get(s.name) ?? [] held: holds.get(s.name) ?? []
}; };
@@ -44,11 +43,10 @@ export function fleetSnapshot(serverName?: string) {
serverNow: Date.now(), serverNow: Date.now(),
reservedNames: serverName === undefined ? (db().prepare("SELECT j.server FROM server_creation_jobs j JOIN server_deployments d ON d.id = j.deployment_id WHERE d.state IN ('queued', 'running', 'awaiting-health')").all() as { server: string }[]).map(row => row.server) : [], reservedNames: serverName === undefined ? (db().prepare("SELECT j.server FROM server_creation_jobs j JOIN server_deployments d ON d.id = j.deployment_id WHERE d.state IN ('queued', 'running', 'awaiting-health')").all() as { server: string }[]).map(row => row.server) : [],
integrations: { integrations: {
native: hosts.length > 0 || !komodoConfigured(), native: true,
hostsReady: hosts.filter(host => host.readiness === 'ready').length, hostsReady: hosts.filter(host => host.readiness === 'ready').length,
hostsTotal: hosts.length, hostsTotal: hosts.length,
rcon: rconConfigured(), rcon: rconConfigured(),
komodo: komodoConfigured(),
rconTimeoutMs: config.rcon.timeoutMs, rconTimeoutMs: config.rcon.timeoutMs,
defaultRconHost: rconHost defaultRconHost: rconHost
} }
+1 -1
View File
@@ -59,7 +59,7 @@ export async function probePlayerEvidence(server: string): Promise<PlayerEvidenc
return { players: null, source: null, observedAt: new Date().toISOString() }; return { players: null, source: null, observedAt: new Date().toISOString() };
} }
/** Komodo readiness remains a caller prerequisite. Old-process telemetry is excluded. */ /** Host agent readiness remains a caller prerequisite. Old-process telemetry is excluded. */
export async function probeServerStatus(server: string, after?: string | null): Promise< export async function probeServerStatus(server: string, after?: string | null): Promise<
{ ok: true; players: number; map: string | null; source: 'rcon' | 'telemetry'; observedAt: string; receivedAt?: string } | { ok: false; reason: string; observedAt: string } { ok: true; players: number; map: string | null; source: 'rcon' | 'telemetry'; observedAt: string; receivedAt?: string } | { ok: false; reason: string; observedAt: string }
> { > {
+1 -2
View File
@@ -7,7 +7,6 @@ import { telemetryReport, latestGameSubmission } from './status-evidence.js';
import { latestGameSubmissions } from './lifecycle-evidence.js'; import { latestGameSubmissions } from './lifecycle-evidence.js';
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import { parseGameEvent, type GameTelemetry } from '../../telemetry.js'; import { parseGameEvent, type GameTelemetry } from '../../telemetry.js';
import { restartRequestInFlight } from '../provisioning/restarts.js';
interface Evidence { provisioning: boolean; settling: boolean; telemetry: GameTelemetry | null; submission: string | null } interface Evidence { provisioning: boolean; settling: boolean; telemetry: GameTelemetry | null; submission: string | null }
@@ -33,7 +32,7 @@ export function observationReader() {
const operation = operations.get(server.name), restart = restarts.get(server.name); const operation = operations.get(server.name), restart = restarts.get(server.name);
return serverObservation(server, { return serverObservation(server, {
provisioning: !!operation && ['create','rebuild','start'].includes(operation.action) && ['queued','running','awaiting-health'].includes(operation.state), provisioning: !!operation && ['create','rebuild','start'].includes(operation.action) && ['queued','running','awaiting-health'].includes(operation.state),
settling: restartRequestInFlight(server.name) || !!(restart && !restart.healthy_at && Date.now() - Date.parse(restart.requested_at) < 15_000), settling: !!(restart && !restart.healthy_at && Date.now() - Date.parse(restart.requested_at) < 15_000),
telemetry: reports.get(server.name) ?? null, submission: submissions.get(server.name) ?? null telemetry: reports.get(server.name) ?? null, submission: submissions.get(server.name) ?? null
}); });
}; };
+1 -1
View File
@@ -32,7 +32,7 @@ export async function readCapacityCurrent(query: CapacityQuery): Promise<Capacit
if (host.decommission) reasons.push('Host is being removed.'); if (host.decommission) reasons.push('Host is being removed.');
if (!host.credentialActive && host.enrolled) reasons.push('Host credentials are inactive.'); if (!host.credentialActive && host.enrolled) reasons.push('Host credentials are inactive.');
if (host.readiness !== 'ready') reasons.push(...host.readinessReasons); if (host.readiness !== 'ready') reasons.push(...host.readinessReasons);
if (!host.inventory.runtimeImages?.includes(config.komodo.defaultImage)) reasons.push('The default runtime is not ready.'); if (!host.inventory.runtimeImages?.includes(config.defaultRuntimeImage)) reasons.push('The default runtime is not ready.');
if (!requestedGames.some(game => games.includes(game))) reasons.push('Latest game files for this selection are not ready.'); if (!requestedGames.some(game => games.includes(game))) reasons.push('Latest game files for this selection are not ready.');
if ((host.inventory.storageAvailable ?? 0) < host.minimumFreeBytes + 1024 ** 3) reasons.push('Storage is below the launch reserve.'); if ((host.inventory.storageAvailable ?? 0) < host.minimumFreeBytes + 1024 ** 3) reasons.push('Storage is below the launch reserve.');
const eligible = connected && host.readiness === 'ready' && reasons.length === 0; const eligible = connected && host.readiness === 'ready' && reasons.length === 0;
+1 -1
View File
@@ -88,7 +88,7 @@ function targets(host: Host): Target[] {
&& operation.sequence > enabling.operation_floor && operation.state === 'completed' && operation.sequence > enabling.operation_floor && operation.state === 'completed'
&& Date.parse(host.lastHeartbeat ?? '') > Date.parse(operation.updated_at); && Date.parse(host.lastHeartbeat ?? '') > Date.parse(operation.updated_at);
const selections = queuedSelections(host.id); const selections = queuedSelections(host.id);
const images = new Set([config.komodo.defaultImage, ...selections.flatMap(selection => selection.image ? [selection.image] : [])]); const images = new Set([config.defaultRuntimeImage, ...selections.flatMap(selection => selection.image ? [selection.image] : [])]);
const result: Target[] = [...images].map(image => { const result: Target[] = [...images].map(image => {
const operation = latestOperation(host.id, 'runtime.pull', 'image', image); const operation = latestOperation(host.id, 'runtime.pull', 'image', image);
const runtimeReady = host.inventory.runtimeImages?.includes(image) && verifiedThisEnable(operation); const runtimeReady = host.inventory.runtimeImages?.includes(image) && verifiedThisEnable(operation);
-832
View File
@@ -1,832 +0,0 @@
import { config, komodoConfigured } from '../config.js';
import { getServer } from '../fleet/servers.js';
import { nativeHealth, nativeInstance } from '../backends/native.js';
// Ports `komodo_restart` in push_scripts.sh:622. Three shapes depending on
// the tracked komodo_type: deployment | container | stack.
export type KomodoRestartResult =
| { ok: true }
| { ok: false; kind: 'unconfigured' | 'no-name' | 'http-error' | 'unreachable'; message: string };
async function komodoCall(path: string, body: unknown): Promise<Response> {
const url = `${config.komodo.address.replace(/\/+$/, '')}/${path}`;
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), config.httpTimeoutMs);
try {
return await fetch(url, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-Api-Key': config.komodo.apiKey,
'X-Api-Secret': config.komodo.apiSecret
},
body: JSON.stringify(body),
signal: controller.signal
});
} finally {
clearTimeout(timer);
}
}
export async function komodoRestart(serverName: string): Promise<KomodoRestartResult> {
if (!komodoConfigured()) {
return { ok: false, kind: 'unconfigured', message: 'komodo not configured' };
}
const row = getServer(serverName);
if (!row?.komodo_name) return { ok: false, kind: 'no-name', message: 'no komodo_name set for server' };
const type = row.komodo_type ?? 'deployment';
let path: string;
let body: Record<string, string>;
switch (type) {
case 'stack':
path = 'execute/RestartStack';
body = { stack: row.komodo_name };
break;
case 'container':
if (!row.komodo_server) return { ok: false, kind: 'no-name', message: 'container type needs komodo_server' };
path = 'execute/RestartContainer';
body = { server: row.komodo_server, container: row.komodo_name };
break;
case 'deployment':
default:
path = 'execute/RestartDeployment';
body = { deployment: row.komodo_name };
break;
}
const url = `${config.komodo.address.replace(/\/+$/, '')}/${path}`;
let res: Response;
try {
res = await komodoCall(path, body);
} catch (err) {
return { ok: false, kind: 'unreachable', message: describeFetchError(err, url) };
}
if (!res.ok) {
return { ok: false, kind: 'http-error', message: `HTTP ${res.status}` };
}
// The script also checks the response body: if it's an object with
// .success === false, treat as failure. If .success is absent, trust the
// HTTP status (line 646).
try {
const json = await res.json();
if (json && typeof json === 'object' && 'success' in json && json.success === false) {
return { ok: false, kind: 'http-error', message: 'komodo returned success=false' };
}
} catch {
// Non-JSON body is fine — HTTP 2xx already indicates success.
}
return { ok: true };
}
// ---------- Port resolution ----------
export type KomodoPortFailure =
| 'unconfigured'
| 'no-name'
| 'unreachable' // network / DNS / timeout
| 'http-error' // non-2xx
| 'unparseable' // 2xx but body wasn't JSON
| 'no-udp-in-response'; // parsed but no UDP port found
export interface KomodoPortSuccess {
ok: true;
port: number;
snippet: string;
url: string;
}
export interface KomodoPortFailureResult {
ok: false;
reason: KomodoPortFailure;
detail: string;
httpStatus?: number;
snippet?: string;
// URL we tried (or would have tried) — surfaced in the UI so operators
// can `curl` it from the container to verify reachability.
url?: string;
}
export type KomodoPortResult = KomodoPortSuccess | KomodoPortFailureResult;
const SNIPPET_LIMIT = 4000;
function truncate(s: string): string {
if (s.length <= SNIPPET_LIMIT) return s;
return s.slice(0, SNIPPET_LIMIT) + `\n… (${s.length - SNIPPET_LIMIT} more bytes)`;
}
/**
* Turn a Node/Undici fetch failure into something an operator can act on.
* The default `.message` for an aborted fetch is just "This operation was
* aborted" — useless. For network failures Node hides the real cause under
* `.cause` (e.g. `ECONNREFUSED`, `ENOTFOUND`, `EHOSTUNREACH`). Unwrap both
* and always include the URL we tried.
*/
function describeFetchError(err: unknown, url: string): string {
const e = err as (Error & { name?: string; cause?: unknown }) | undefined;
if (e?.name === 'AbortError') {
return `timed out after ${config.httpTimeoutMs}ms dialing ${url}`;
}
const cause = e?.cause as (Error & { code?: string }) | undefined;
if (cause && (cause.code || cause.message)) {
const code = cause.code ? `${cause.code}: ` : '';
return `${code}${cause.message ?? 'network error'} (dialing ${url})`;
}
return `${e?.message ?? String(err)} (dialing ${url})`;
}
function safeJsonStringify(value: unknown): string {
try {
return JSON.stringify(value, null, 2);
} catch {
return String(value);
}
}
/**
* Detailed variant of `komodoResolveGamePort` — returns a discriminated
* union so callers can distinguish "network died" from "auth failed" from
* "we parsed the JSON but couldn't find a UDP port in a shape we recognize".
*
* On success, the response snippet is included so the caller can render it
* as "here's where we found the port". On no-udp-in-response, the snippet
* is the only way we'll know what shape Komodo actually emitted — surface
* it prominently in the UI so we can extend `extractUdpPort` for that shape.
*/
export async function komodoResolvePortDetailed(serverName: string): Promise<KomodoPortResult> {
if (!komodoConfigured()) {
return { ok: false, reason: 'unconfigured', detail: 'KOMODO_ADDRESS/KEY/SECRET are not all set' };
}
const row = getServer(serverName);
if (!row?.komodo_name) {
return { ok: false, reason: 'no-name', detail: `server has no komodo_name: ${serverName}` };
}
const type = row.komodo_type ?? 'deployment';
let path: string;
let body: Record<string, string>;
switch (type) {
case 'stack':
path = 'read/GetStack';
body = { stack: row.komodo_name };
break;
case 'container':
if (!row.komodo_server) {
return { ok: false, reason: 'no-name', detail: 'container type needs komodo_server' };
}
path = 'read/InspectContainer';
body = { server: row.komodo_server, container: row.komodo_name };
break;
case 'deployment':
default:
path = 'read/GetDeployment';
body = { deployment: row.komodo_name };
break;
}
const url = `${config.komodo.address.replace(/\/+$/, '')}/${path}`;
let res: Response;
try {
res = await komodoCall(path, body);
} catch (err) {
return { ok: false, reason: 'unreachable', detail: describeFetchError(err, url), url };
}
// Read the body up front — we want the snippet even on non-2xx so the
// UI can show what the error page actually said.
const rawText = await res.text().catch(() => '');
if (!res.ok) {
return {
ok: false,
reason: 'http-error',
detail: `HTTP ${res.status} ${res.statusText || ''}`.trim(),
httpStatus: res.status,
snippet: truncate(rawText),
url
};
}
let json: unknown;
try {
json = JSON.parse(rawText);
} catch (err) {
return {
ok: false,
reason: 'unparseable',
detail: `response body is not JSON: ${(err as Error).message}`,
httpStatus: res.status,
snippet: truncate(rawText),
url
};
}
const port = extractUdpPort(json);
const snippet = truncate(safeJsonStringify(json));
if (port === null) {
return {
ok: false,
reason: 'no-udp-in-response',
detail: 'parsed the response but no UDP port matched any known shape',
httpStatus: res.status,
snippet,
url
};
}
return { ok: true, port, snippet, url };
}
/**
* Query Komodo's read API for a server's UDP game port. Called lazily by
* the RCON client and cached into `servers.game_port` on success. Thin shim
* over `komodoResolvePortDetailed` that swallows diagnostics — use the
* detailed variant when you care why it failed.
*/
export async function komodoResolveGamePort(serverName: string): Promise<number | null> {
const result = await komodoResolvePortDetailed(serverName);
return result.ok ? result.port : null;
}
// ---------- Health probe ----------
export type KomodoHealthResult =
| { ok: true; state: string }
| { ok: false; reason: string; state?: string; snippet?: string };
// Komodo's container summary has { state: "running", container: { status:
// "Up 35 seconds (health: starting)" } }. Inspect responses instead have
// State.Health.Status (or snake/lowercase equivalents). Runtime health wins
// over a coarse running flag. Unknown/configuration strings are not readiness.
export function extractContainerState(payload: unknown): string | null {
const seen = new Set<unknown>();
const known = new Set(['running','healthy','starting','restarting','unhealthy','stopped','exited','created','dead','paused','removing','notdeployed','unknown']);
const stateOf = (value: unknown): string | null => {
if (typeof value !== 'string') return null;
const state = value.toLowerCase().trim();
if (known.has(state)) return state;
if (/^up\b/.test(state)) {
const health = state.match(/\((?:health:\s*)?(starting|unhealthy|healthy)\)/)?.[1];
return health && health !== 'healthy' ? health : 'running';
}
if (/^exited\b/.test(state)) return 'exited';
if (/^restarting\b/.test(state)) return 'restarting';
return null;
};
const walk = (value: unknown): string | null => {
if (!value || typeof value !== 'object' || seen.has(value)) return null;
seen.add(value);
if (Array.isArray(value)) {
const states = value.map(walk).filter((state): state is string => !!state);
return states.find(state => !['running','healthy'].includes(state)) ?? states[0] ?? null;
}
const object = value as Record<string, unknown>;
const rawState = object.State ?? object.state;
const state = stateOf(rawState) ?? stateOf(object.container_state ?? object.containerState ?? object.deployment_state ?? object.deploymentState);
// A stopped process is never ready just because its previous health check passed.
if (state && !['running','healthy','unknown'].includes(state)) return state;
if (rawState && typeof rawState === 'object') {
const nested = walk(rawState);
if (nested) return nested;
}
for (const key of ['container','Container','containers','Containers']) {
if (object[key] === null && state === 'running') return 'unknown';
const nested = walk(object[key]);
if (nested) return nested;
}
const status = stateOf(object.Status ?? object.status ?? object.docker_status);
if (status && !['running','healthy'].includes(status)) return status;
const health = object.Health ?? object.health;
const healthState = stateOf(typeof health === 'object' && health ?
(health as Record<string, unknown>).Status ?? (health as Record<string, unknown>).status : health);
if (healthState && healthState !== 'healthy') return healthState;
if (state || status) return state ?? status;
if (object.Running === true || object.running === true) return healthState ?? 'running';
for (const [key, item] of Object.entries(object)) {
if (['config','Config','labels','Labels'].includes(key)) continue;
const nested = walk(item);
if (nested) return nested;
}
return null;
};
return walk(payload);
}
const HEALTHY_STATES = new Set(['running', 'healthy', 'ok', 'started', 'up']);
async function fetchJsonWithSnippet(
path: string,
body: Record<string, string>
): Promise<{ ok: true; json: unknown; snippet: string } | { ok: false; reason: string; snippet?: string; unsupportedEndpoint?: boolean }> {
const url = `${config.komodo.address.replace(/\/+$/, '')}/${path}`;
let res: Response;
try {
res = await komodoCall(path, body);
} catch (err) {
return { ok: false, reason: describeFetchError(err, url) };
}
const text = await res.text().catch(() => '');
if (!res.ok) {
const unsupportedEndpoint = res.status === 404 || (
[400, 405, 422, 500, 501].includes(res.status)
&& /\b(?:unknown|unsupported|unrecognized)\s+(?:method|endpoint|variant)\b|\bmethod not found\b/i.test(text)
);
return { ok: false, reason: `HTTP ${res.status}`, snippet: truncate(text), unsupportedEndpoint };
}
let json: unknown;
try {
json = JSON.parse(text);
} catch (err) {
return { ok: false, reason: `unparseable json: ${(err as Error).message}`, snippet: truncate(text) };
}
return { ok: true, json, snippet: truncate(safeJsonStringify(json)) };
}
/**
* Ask Komodo whether the server's container is currently running. Used by the
* sweep during the health phase (post-restart) and by the beta runner. For
* deployment-type servers we hit the container-inspect endpoint first because
* GetDeployment returns config, not runtime state — falling back only when
* the container endpoint isn't available.
*/
export async function komodoInspectHealthy(serverName: string): Promise<KomodoHealthResult> {
if (getServer(serverName)?.native_instance_id) return nativeHealth(serverName);
if (!komodoConfigured()) return { ok: false, reason: 'komodo not configured' };
const row = getServer(serverName);
if (!row?.komodo_name) return { ok: false, reason: 'no komodo_name for server' };
const type = row.komodo_type ?? 'deployment';
const attempts: Array<{ path: string; body: Record<string, string> }> = [];
switch (type) {
case 'stack':
attempts.push({ path: 'read/GetStack', body: { stack: row.komodo_name } });
break;
case 'container':
if (!row.komodo_server) return { ok: false, reason: 'container type needs komodo_server' };
attempts.push({
path: 'read/InspectContainer',
body: { server: row.komodo_server, container: row.komodo_name }
});
break;
case 'deployment':
default:
// Container-specific endpoints first — they return docker-inspect
// shape with State.Status. GetDeployment is the last-resort
// config-only fallback.
attempts.push({ path: 'read/GetDeploymentContainer', body: { deployment: row.komodo_name } });
attempts.push({ path: 'read/InspectDeploymentContainer', body: { deployment: row.komodo_name } });
attempts.push({ path: 'read/GetDeploymentStatus', body: { deployment: row.komodo_name } });
attempts.push({ path: 'read/GetDeployment', body: { deployment: row.komodo_name } });
break;
}
let lastSnippet: string | undefined;
for (const { path, body } of attempts) {
const r = await fetchJsonWithSnippet(path, body);
if (!r.ok) {
// Alternate endpoints handle API-version differences. Retrying an
// outage against every endpoint delays all other server probes.
if (!r.unsupportedEndpoint) return { ok: false, reason: r.reason, snippet: r.snippet };
lastSnippet = r.snippet ?? lastSnippet;
continue;
}
const state = extractContainerState(r.json);
if (state) {
const healthy = HEALTHY_STATES.has(state.toLowerCase());
if (!healthy) return { ok: false, state, reason: `container state: ${state}`, snippet: r.snippet };
return { ok: true, state };
}
lastSnippet = r.snippet;
}
return {
ok: false,
reason: 'no container state found in any known Komodo response shape',
snippet: lastSnippet
};
}
// ---------- Write / Execute (create + destroy deployments) ----------
export type KomodoWriteResult =
| { ok: true }
| { ok: false; kind: 'unconfigured' | 'http-error' | 'unreachable'; message: string; url?: string; httpStatus?: number };
/**
* Env var pairs get flattened into a single newline-delimited string, matching
* Komodo's DeploymentConfig.environment shape.
*/
export interface DeploymentEnvVar {
variable: string;
value: string;
}
export interface CreateDeploymentSpec {
name: string; // Komodo deployment name (== EC server name)
server: string; // Komodo server (host) id or name
image: string; // 'ghcr.io/ayymoss/plutainer:latest'
network?: string; // 'komodo_default'
ports: string[]; // ['28960:28960/udp', '28960:28960/tcp']
volumes: string[]; // ['/host/path:/container/path', 'namedvol:/container/path:ro']
environment: DeploymentEnvVar[];
}
async function postAndClassify(path: string, body: unknown): Promise<KomodoWriteResult> {
const url = `${config.komodo.address.replace(/\/+$/, '')}/${path}`;
let res: Response;
try {
res = await komodoCall(path, body);
} catch (err) {
return { ok: false, kind: 'unreachable', message: describeFetchError(err, url), url };
}
if (!res.ok) {
const snippet = await res.text().catch(() => '');
return {
ok: false,
kind: 'http-error',
httpStatus: res.status,
message: `HTTP ${res.status}${snippet ? ` — ${truncate(snippet)}` : ''}`,
url
};
}
// Some Komodo endpoints return `{ success: false }` with a 200. Treat that
// as a failure so callers can surface the real problem.
try {
const json = await res.json();
if (json && typeof json === 'object' && 'success' in json && json.success === false) {
const msg = typeof (json as { message?: unknown }).message === 'string'
? (json as { message: string }).message
: 'komodo returned success=false';
return { ok: false, kind: 'http-error', message: msg, url, httpStatus: res.status };
}
} catch {
// Non-JSON body — trust the 2xx.
}
return { ok: true };
}
/**
* Create a new deployment entity in Komodo. Doesn't start the container —
* caller must follow up with `komodoDeploy(name)` to actually run it.
*
* The config shape here targets Komodo's DeploymentConfig: newline-delimited
* ports/volumes strings and an environment array with `{ variable, value }`
* entries. If Komodo rejects the payload, the error surfaces in the http-error
* message so we can iterate on the shape without guessing.
*/
export async function komodoCreateDeployment(spec: CreateDeploymentSpec): Promise<KomodoWriteResult> {
if (!komodoConfigured()) {
return { ok: false, kind: 'unconfigured', message: 'komodo not configured' };
}
const body = {
name: spec.name,
config: {
server_id: spec.server,
image: { type: 'Image', params: { image: spec.image } },
network: spec.network ?? 'komodo_default',
ports: spec.ports.join('\n'),
volumes: spec.volumes.join('\n'),
environment: spec.environment,
restart: 'unless-stopped'
}
};
return postAndClassify('write/CreateDeployment', body);
}
/** Start (or restart) the container backing a Komodo deployment. */
export async function komodoDeploy(deployment: string): Promise<KomodoWriteResult> {
if (!komodoConfigured()) {
return { ok: false, kind: 'unconfigured', message: 'komodo not configured' };
}
return postAndClassify('execute/Deploy', { deployment });
}
/** Runtime controls use the saved Komodo resource, never the EC display name. */
export async function komodoRuntimeAction(server: string, action: 'start' | 'stop'): Promise<KomodoRestartResult> {
if (!komodoConfigured()) return { ok: false, kind: 'unconfigured', message: 'komodo not configured' };
const row = getServer(server);
if (!row?.komodo_name) return { ok: false, kind: 'no-name', message: 'no komodo_name set for server' };
const type = row.komodo_type ?? 'deployment';
const verb = action === 'start' ? 'Start' : 'Stop';
if (type === 'container') {
if (!row.komodo_server) return { ok: false, kind: 'no-name', message: 'container type needs komodo_server' };
return postAndClassify(`execute/${verb}Container`, { server: row.komodo_server, container: row.komodo_name });
}
if (type === 'stack') return postAndClassify(`execute/${verb}Stack`, { stack: row.komodo_name });
return postAndClassify(`execute/${verb}Deployment`, { deployment: row.komodo_name });
}
/** Stop + remove the container backing a Komodo deployment. Entity remains. */
export async function komodoDestroyDeployment(deployment: string): Promise<KomodoWriteResult> {
if (!komodoConfigured()) {
return { ok: false, kind: 'unconfigured', message: 'komodo not configured' };
}
return postAndClassify('execute/DestroyDeployment', { deployment });
}
/** Remove the deployment entity from Komodo entirely. */
export async function komodoDeleteDeployment(deployment: string): Promise<KomodoWriteResult> {
if (!komodoConfigured()) {
return { ok: false, kind: 'unconfigured', message: 'komodo not configured' };
}
// Komodo's DeleteDeployment historically accepts `{ id }` where id may be
// the name; supply both to be forgiving.
return postAndClassify('write/DeleteDeployment', { id: deployment, name: deployment });
}
// ---------- Tags ----------
export type KomodoTagResult =
| { ok: true; id: string }
| { ok: false; message: string };
// Accept API string IDs and raw Mongo Extended JSON: { _id: { $oid: "..." } }.
function extractTagId(t: unknown): string | null {
if (!t || typeof t !== 'object') return null;
const o = t as { id?: unknown; _id?: unknown };
if (typeof o.id === 'string' && o.id.length > 0) return o.id;
if (typeof o._id === 'string' && o._id.length > 0) return o._id;
if (o._id && typeof o._id === 'object') {
const oid = (o._id as { $oid?: unknown }).$oid;
if (typeof oid === 'string' && oid.length > 0) return oid;
}
return null;
}
// ListTags response shape varies by Komodo version: bare array `[...]`, or
// wrapped under one of a few common keys. Peel back the common wrappers.
function unwrapTagList(body: unknown): unknown[] {
if (Array.isArray(body)) return body;
if (body && typeof body === 'object') {
for (const key of ['tags', 'data', 'results', 'items']) {
const val = (body as Record<string, unknown>)[key];
if (Array.isArray(val)) return val;
}
}
return [];
}
/**
* Fetch the tag list from Komodo. Two request styles are attempted in order:
*
* 1. `POST /read/ListTags` with an empty body — the per-endpoint style
* already used by the rest of this file (see execute/RestartDeployment).
* 2. `POST /read` with `{type: "ListTags", params: {}}` — the tagged-union
* style used by mbecker20/komodo's canonical API. If your Komodo build
* wires up only the tagged-union path, style #1 returns something we
* can't parse (empty array or non-tag payload) and we fall through.
*
* The winning parsed list is returned. The raw response bodies get logged
* only when both attempts fail to yield a match, so a fresh Komodo API shape
* is diagnosable without spamming stderr on the happy path.
*/
async function fetchTagsWithFallback(): Promise<{ tags: unknown[]; rawBodies: string[] }> {
const attempts: Array<{ path: string; body: unknown }> = [
{ path: 'read/ListTags', body: {} },
{ path: 'read', body: { type: 'ListTags', params: {} } }
];
const rawBodies: string[] = [];
for (const attempt of attempts) {
let res: Response;
try {
res = await komodoCall(attempt.path, attempt.body);
} catch (err) {
rawBodies.push(`${attempt.path}: network error — ${(err as Error).message}`);
continue;
}
if (!res.ok) {
const snippet = await res.text().catch(() => '');
rawBodies.push(`${attempt.path}: HTTP ${res.status}${snippet ? ` — ${truncate(snippet)}` : ''}`);
continue;
}
const raw = await res.text().catch(() => '');
rawBodies.push(`${attempt.path}: ${truncate(raw)}`);
let parsed: unknown;
try {
parsed = JSON.parse(raw);
} catch {
continue;
}
const tags = unwrapTagList(parsed);
if (tags.length > 0) return { tags, rawBodies };
}
return { tags: [], rawBodies };
}
async function findTagByName(name: string): Promise<KomodoTagResult | null> {
const { tags, rawBodies } = await fetchTagsWithFallback();
for (const t of tags) {
if (!t || typeof t !== 'object') continue;
const tagName = (t as { name?: unknown }).name;
if (typeof tagName !== 'string' || tagName !== name) continue;
const id = extractTagId(t);
if (id) return { ok: true, id };
}
// Miss — dump the raw bodies so the next iteration knows the real shape.
console.warn(
`[komodoEnsureTag] no matchable tag "${name}" in any ListTags variant. Bodies:\n ` +
rawBodies.join('\n ')
);
return null;
}
/**
* Resolve a tag by name, creating it if it doesn't exist. Returns the tag's
* Komodo id (Mongo ObjectId string) for use in UpdateResourceMeta.
*
* Best-effort: callers should treat failure as "skip tagging", not "roll back
* the whole operation". Tags are cosmetic organization inside Komodo.
*/
export async function komodoEnsureTag(name: string): Promise<KomodoTagResult> {
if (!komodoConfigured()) return { ok: false, message: 'komodo not configured' };
// 1. Look for an existing tag by name.
const existing = await findTagByName(name);
if (existing) return existing;
// 2. Not found — create it.
let createRes: Response;
try {
createRes = await komodoCall('write/CreateTag', { name });
} catch (err) {
return { ok: false, message: `CreateTag ${name}: ${(err as Error).message}` };
}
if (createRes.ok) {
const body = await createRes.json().catch(() => null);
const id = extractTagId(body);
if (id) return { ok: true, id };
// 200 but shape we don't recognize — fall back to re-list rather than
// fail, since the tag definitely exists now.
const fresh = await findTagByName(name);
if (fresh) return fresh;
console.warn(`[komodoEnsureTag] CreateTag ${name}: unexpected 2xx body, re-list also missed`, body);
return { ok: false, message: `CreateTag ${name}: unexpected response shape` };
}
// Non-2xx. Duplicate-key 500 means the tag exists under a different
// ListTags projection than we probed — re-list and try to find it. If
// that still misses, surface the create error.
const snippet = await createRes.text().catch(() => '');
const looksLikeDuplicate =
createRes.status === 500 &&
(/duplicate key/i.test(snippet) || /E11000/i.test(snippet) || /dup key/i.test(snippet));
if (looksLikeDuplicate) {
const fresh = await findTagByName(name);
if (fresh) return fresh;
return {
ok: false,
message: `CreateTag ${name}: duplicate reported but re-list missed it. Body: ${truncate(snippet)}`
};
}
return {
ok: false,
message: `CreateTag ${name}: HTTP ${createRes.status}${snippet ? ` — ${truncate(snippet)}` : ''}`
};
}
/**
* Attach tags to a Komodo resource. `tagIds` is the array of tag Mongo IDs
* from `komodoEnsureTag`. Replaces the resource's existing tag set — pass the
* full desired list, not a delta.
*
* Komodo's target.id accepts either the true id or the resource name for
* Deployments, so we pass the name here to save a lookup.
*/
export async function komodoUpdateTagsOnResource(
targetType: 'Deployment' | 'Stack' | 'Server' | 'Repo' | 'Build' | 'Procedure',
targetName: string,
tagIds: string[]
): Promise<KomodoWriteResult> {
if (!komodoConfigured()) {
return { ok: false, kind: 'unconfigured', message: 'komodo not configured' };
}
// UpdateResourceMeta owns tags; omit the other optional metadata fields
// so the deployment's description and template flag remain unchanged.
return postAndClassify('write/UpdateResourceMeta', {
target: { type: targetType, id: targetName },
tags: tagIds
});
}
// ---------- Extractor ----------
// Property names Komodo/Docker use for the collection of port mappings.
const PORT_ARRAY_KEYS = new Set(['ports', 'Ports', 'port_bindings', 'PortBindings']);
// Value field names for the host-facing port on an object entry.
const HOST_PORT_KEYS = ['published', 'PublishedPort', 'public_port', 'PublicPort', 'HostPort', 'hostPort', 'host_port'];
interface PortsCandidate {
arr: unknown[];
/** True when the array was reached via a `ports`-flavored key — preferred. */
preferred: boolean;
}
/**
* Walks a Komodo response looking for a UDP port. Handles the shapes we've
* seen in the wild:
*
* docker-compose object: [{ protocol: "udp", published: 4976, target: 28960 }]
* docker-compose caps: [{ Protocol: "udp", PublishedPort: 4976 }]
* docker-run string: ["4976:28960/udp", "3074:3074/tcp"]
* host-network string: ["28962/udp"]
* docker-inspect object: { "28960/udp": [{ "HostIp": "0.0.0.0", "HostPort": "28962" }] }
*
* Arrays reached via a `ports`/`Ports`/`port_bindings`/`PortBindings` key
* are preferred over arrays discovered elsewhere in the tree, so an
* unrelated array (e.g. tags, labels) can't shadow a real port list.
*/
export function extractUdpPort(payload: unknown): number | null {
const candidates: PortsCandidate[] = [];
// Objects whose keys encode `<container-port>/<proto>` — a la docker inspect.
const containerKeyed: Record<string, unknown>[] = [];
const seen = new Set<unknown>();
const walk = (v: unknown, viaPortsKey: boolean): void => {
if (v === null || typeof v !== 'object' || seen.has(v)) return;
seen.add(v);
if (Array.isArray(v)) {
for (const item of v) walk(item, false);
candidates.push({ arr: v, preferred: viaPortsKey });
return;
}
const obj = v as Record<string, unknown>;
// Detect docker-inspect container-keyed objects: any key like `28960/udp`.
for (const k of Object.keys(obj)) {
if (/^\d+\/(udp|tcp)$/i.test(k)) {
containerKeyed.push(obj);
break;
}
}
for (const [key, val] of Object.entries(obj)) {
// Komodo's GetDeployment renders `config.ports` as a single
// newline-delimited string ("28963:28963/udp\n28963:28963/tcp").
// Synthesize an array from it and treat as a preferred candidate.
if (PORT_ARRAY_KEYS.has(key) && typeof val === 'string' && val.trim()) {
const parts = val
.split(/[\r\n,;]+/)
.map((s) => s.trim())
.filter(Boolean);
if (parts.length > 0) {
candidates.push({ arr: parts, preferred: true });
}
}
walk(val, PORT_ARRAY_KEYS.has(key));
}
};
walk(payload, false);
// Sort so preferred `ports`-keyed arrays win over accidental hits.
candidates.sort((a, b) => (a.preferred === b.preferred ? 0 : a.preferred ? -1 : 1));
// Try object-entry shape first — most Komodo/Docker responses use this.
for (const { arr } of candidates) {
for (const entry of arr) {
if (entry && typeof entry === 'object' && !Array.isArray(entry)) {
const e = entry as Record<string, unknown>;
const proto = String(e.protocol ?? e.Protocol ?? e.type ?? e.Type ?? '').toLowerCase();
if (proto !== 'udp') continue;
for (const field of HOST_PORT_KEYS) {
const raw = e[field];
if (raw === undefined || raw === null) continue;
const num = typeof raw === 'string' ? Number.parseInt(raw, 10) : Number(raw);
if (Number.isFinite(num) && num > 0) return num;
}
}
}
}
// Then string shapes: "host:container/udp" or "container/udp" (host-net).
for (const { arr } of candidates) {
for (const entry of arr) {
if (typeof entry !== 'string') continue;
const hostMapped = /^(\d+):\d+\/udp$/i.exec(entry);
if (hostMapped) return Number.parseInt(hostMapped[1], 10);
const hostNet = /^(\d+)\/udp$/i.exec(entry);
if (hostNet) return Number.parseInt(hostNet[1], 10);
}
}
// Finally, docker-inspect container-keyed shape:
// { "28960/udp": [{ HostIp, HostPort }] }
for (const obj of containerKeyed) {
for (const [key, val] of Object.entries(obj)) {
if (!/^\d+\/udp$/i.test(key)) continue;
if (!Array.isArray(val)) continue;
for (const binding of val) {
if (!binding || typeof binding !== 'object') continue;
const b = binding as Record<string, unknown>;
for (const field of HOST_PORT_KEYS) {
const raw = b[field];
if (raw === undefined || raw === null) continue;
const num = typeof raw === 'string' ? Number.parseInt(raw, 10) : Number(raw);
if (Number.isFinite(num) && num > 0) return num;
}
}
}
}
return null;
}
-93
View File
@@ -1,93 +0,0 @@
import { createConnection } from 'node:net';
import { lookup } from 'node:dns/promises';
export type TcpProbeOutcome =
| 'connected'
| 'refused'
| 'timeout'
| 'dns-fail'
| 'unreachable'
| 'error';
export interface TcpProbeResult {
label: string;
host: string;
port: number;
resolvedIp: string | null;
outcome: TcpProbeOutcome;
elapsedMs: number;
detail?: string;
}
const DEFAULT_TIMEOUT_MS = 3000;
/**
* Best-effort DNS resolution. Returns null on failure so the caller can
* distinguish "DNS failed" from "connect failed".
*/
async function resolve(host: string): Promise<{ ip: string | null; error: string | null }> {
try {
const r = await lookup(host);
return { ip: r.address, error: null };
} catch (err) {
const e = err as NodeJS.ErrnoException;
return { ip: null, error: `${e.code ?? 'UNKNOWN'}: ${e.message}` };
}
}
/**
* Attempt a raw TCP connect to `host:port` with an explicit timeout, and
* classify the result. Purely diagnostic — no bytes are sent after the
* handshake completes. Isolates network reachability from every HTTP-layer
* concern (auth, TLS, redirects, wrong path).
*/
export async function tcpProbe(
label: string,
host: string,
port: number,
timeoutMs: number = DEFAULT_TIMEOUT_MS
): Promise<TcpProbeResult> {
const started = Date.now();
const { ip, error: dnsError } = await resolve(host);
if (!ip) {
return {
label,
host,
port,
resolvedIp: null,
outcome: 'dns-fail',
elapsedMs: Date.now() - started,
detail: dnsError ?? 'DNS lookup returned no address'
};
}
return new Promise<TcpProbeResult>((resolve) => {
const sock = createConnection({ host: ip, port, timeout: timeoutMs });
let settled = false;
const finish = (outcome: TcpProbeOutcome, detail?: string): void => {
if (settled) return;
settled = true;
try { sock.destroy(); } catch { /* already destroyed */ }
resolve({ label, host, port, resolvedIp: ip, outcome, elapsedMs: Date.now() - started, detail });
};
sock.once('connect', () => finish('connected'));
sock.once('timeout', () => finish('timeout', `no SYN-ACK within ${timeoutMs}ms`));
sock.once('error', (err) => {
const code = (err as NodeJS.ErrnoException).code ?? '';
const msg = (err as Error).message ?? String(err);
if (code === 'ECONNREFUSED') return finish('refused', code);
if (code === 'EHOSTUNREACH' || code === 'ENETUNREACH') return finish('unreachable', `${code}: ${msg}`);
return finish('error', `${code || 'unknown'}: ${msg}`);
});
});
}
export async function tcpProbeAll(
trials: Array<{ label: string; host: string; port: number }>,
timeoutMs: number = DEFAULT_TIMEOUT_MS
): Promise<TcpProbeResult[]> {
// Run in parallel — they don't share sockets and the whole point is to
// finish fast even when some probes hit the timeout.
return Promise.all(trials.map((t) => tcpProbe(t.label, t.host, t.port, timeoutMs)));
}
+2 -2
View File
@@ -5,7 +5,7 @@ import { hasActiveProvisioning } from '../fleet/provisioning.js';
import { latestGameSubmission } from '../fleet/status-evidence.js'; import { latestGameSubmission } from '../fleet/status-evidence.js';
import { latestGameSubmissions } from '../fleet/lifecycle-evidence.js'; import { latestGameSubmissions } from '../fleet/lifecycle-evidence.js';
import { rconPlutoniumVersion } from './rcon.js'; import { rconPlutoniumVersion } from './rcon.js';
import { komodoInspectHealthy } from './komodo.js'; import { nativeHealth } from '../backends/native.js';
import { restartRecoveryPending } from '../provisioning/index.js'; import { restartRecoveryPending } from '../provisioning/index.js';
import { parsePlutoniumManifest, PLUTONIUM_MANIFEST_URL } from '../../plutonium.js'; import { parsePlutoniumManifest, PLUTONIUM_MANIFEST_URL } from '../../plutonium.js';
import type { PlutoniumRelease, PlutoniumVersion } from '../../plutonium.js'; import type { PlutoniumRelease, PlutoniumVersion } from '../../plutonium.js';
@@ -60,7 +60,7 @@ export async function probeVersion(server: string, options: { maintenance?: bool
const epoch = latestGameSubmission(server); const epoch = latestGameSubmission(server);
const recovering = recoveryBlocked(); const recovering = recoveryBlocked();
const health = !recovering && original.desired_state !== 'stopped' const health = !recovering && original.desired_state !== 'stopped'
? await komodoInspectHealthy(server).catch(() => ({ ok: false })) : { ok: false }; ? await nativeHealth(server).catch(() => ({ ok: false })) : { ok: false };
if (suppressed()) return null; if (suppressed()) return null;
let result = !health.ok ? { revision: null, error: 'Waiting for a healthy running server.' } let result = !health.ok ? { revision: null, error: 'Waiting for a healthy running server.' }
: await rconPlutoniumVersion(server).catch(() => ({ revision: null, error: 'Version check failed.' })); : await rconPlutoniumVersion(server).catch(() => ({ revision: null, error: 'Version check failed.' }));
+7 -33
View File
@@ -1,9 +1,8 @@
import { createSocket } from 'node:dgram'; import { createSocket } from 'node:dgram';
import type { Socket } from 'node:dgram'; import type { Socket } from 'node:dgram';
import { config, komodoConfigured, rconConfigured } from '../config.js'; import { config, rconConfigured } from '../config.js';
import { getServer, setServerField } from '../fleet/servers.js'; import { getServer } from '../fleet/servers.js';
import { latestGameSubmission } from '../fleet/lifecycle-evidence.js'; import { latestGameSubmission } from '../fleet/lifecycle-evidence.js';
import { komodoResolveGamePort } from './komodo.js';
import { parsePlutoniumRevision } from '../../plutonium.js'; import { parsePlutoniumRevision } from '../../plutonium.js';
import { nativeInstance, nativeRequest } from '../backends/native.js'; import { nativeInstance, nativeRequest } from '../backends/native.js';
@@ -81,27 +80,15 @@ async function sendRcon(host: string, port: number, command: string): Promise<Se
interface EndpointResolution { interface EndpointResolution {
endpoint: RconEndpoint | null; endpoint: RconEndpoint | null;
// Present when endpoint is null — tells the caller why we couldn't dial. reason?: 'unknown-server' | 'no-port';
reason?: 'unknown-server' | 'no-port-no-komodo' | 'no-port-komodo-empty';
} }
/** /** Resolve a saved RCON endpoint without remote resource discovery. */
* Resolve the RCON endpoint (host + port) for a server, filling
* `servers.game_port` from Komodo on first miss.
*/
async function endpointFor(server: string): Promise<EndpointResolution> { async function endpointFor(server: string): Promise<EndpointResolution> {
const row = getServer(server); const row = getServer(server);
if (!row) return { endpoint: null, reason: 'unknown-server' }; if (!row) return { endpoint: null, reason: 'unknown-server' };
let port = row.game_port; if (!row.game_port) return { endpoint: null, reason: 'no-port' };
if (!port) { return { endpoint: { host: row.rcon_host ?? config.rcon.host, port: row.game_port } };
if (!komodoConfigured()) {
return { endpoint: null, reason: 'no-port-no-komodo' };
}
port = await komodoResolveGamePort(server);
if (port) setServerField(server, 'game_port', port);
}
if (!port) return { endpoint: null, reason: 'no-port-komodo-empty' };
return { endpoint: { host: row.rcon_host ?? config.rcon.host, port } };
} }
function stripPrefix(buf: Buffer): string | null { function stripPrefix(buf: Buffer): string | null {
@@ -421,20 +408,7 @@ export async function rconStatusResult(server: string): Promise<RconStatusResult
if (res.reason === 'unknown-server') { if (res.reason === 'unknown-server') {
return { ok: false, endpoint: null, reason: 'no-port', detail: `unknown server: ${server}` }; return { ok: false, endpoint: null, reason: 'no-port', detail: `unknown server: ${server}` };
} }
if (res.reason === 'no-port-no-komodo') { return { ok: false, endpoint: null, reason: 'no-port', detail: 'No game port is registered for this server.' };
return {
ok: false,
endpoint: null,
reason: 'no-port',
detail: 'no game_port set and Komodo is not configured'
};
}
return {
ok: false,
endpoint: null,
reason: 'no-port',
detail: 'no game_port — Komodo did not return one for this server'
};
} }
const ep = res.endpoint; const ep = res.endpoint;
+4 -5
View File
@@ -1,6 +1,5 @@
import { randomUUID } from 'node:crypto'; import { randomUUID } from 'node:crypto';
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import { komodoConfigured } from '../config.js';
import { nativeInstance } from '../backends/native.js'; import { nativeInstance } from '../backends/native.js';
import { getServer, listServers } from '../fleet/servers.js'; import { getServer, listServers } from '../fleet/servers.js';
import { hasActiveProvisioning } from '../fleet/provisioning.js'; import { hasActiveProvisioning } from '../fleet/provisioning.js';
@@ -100,7 +99,7 @@ export function resolveRecoveredHealthOperation(operationId: string, players: nu
db().prepare('UPDATE server_deployments SET checked_at = ?, players = ? WHERE id = ?').run(now, players, operationId); db().prepare('UPDATE server_deployments SET checked_at = ?, players = ? WHERE id = ?').run(now, players, operationId);
recordProgress(operationId, { blocker: null, observedAt: now, recordProgress(operationId, { blocker: null, observedAt: now,
recovery: { state: 'released', at: now, operationId, message }, recovery: { state: 'released', at: now, operationId, message },
checks: [{ key: 'container', label: 'Container readiness', source: 'komodo', result: 'pass', checkedAt: now }, checks: [{ key: 'container', label: 'Container readiness', source: 'agent', result: 'pass', checkedAt: now },
{ key: 'game', label: 'Game status', source, result: 'pass', actual: players, checkedAt: now }] }); { key: 'game', label: 'Game status', source, result: 'pass', actual: players, checkedAt: now }] });
if (operation.state === 'awaiting-health') { if (operation.state === 'awaiting-health') {
deploymentStage(operationId, message); deploymentStage(operationId, message);
@@ -275,7 +274,7 @@ async function scanServer(server: string, forceScan: boolean): Promise<void> {
if (observation.identity !== monitoringIdentity(server)) return; if (observation.identity !== monitoringIdentity(server)) return;
if (!scanSuppressed(server, 'health') && observation.healthy) { if (!scanSuppressed(server, 'health') && observation.healthy) {
const current = active(server, 'health'); const current = active(server, 'health');
if (current) transition(current.id, 'resolved', 'Fresh game status and Komodo health verified'); if (current) transition(current.id, 'resolved', 'Fresh game status and host agent health verified');
} else if (!scanSuppressed(server, 'health')) { } else if (!scanSuppressed(server, 'health')) {
const problem = detect(server, 'health', observation.reason, observation); const problem = detect(server, 'health', observation.reason, observation);
const startup = ['starting','restarting'].includes(observation.containerState ?? ''); const startup = ['starting','restarting'].includes(observation.containerState ?? '');
@@ -334,8 +333,8 @@ async function run(forceScan: boolean): Promise<void> {
const lastScan = setting('watchdog_last_scan'); const lastScan = setting('watchdog_last_scan');
if (!forceScan && lastScan && Date.now() - Date.parse(lastScan) < WATCHDOG_SCAN_MS) return; if (!forceScan && lastScan && Date.now() - Date.parse(lastScan) < WATCHDOG_SCAN_MS) return;
setSetting('watchdog_last_scan', iso()); setSetting('watchdog_last_scan', iso());
if (!komodoConfigured() && !listServers().some(server => nativeInstance(server.name))) { if (!listServers().some(server => nativeInstance(server.name))) {
setSetting('watchdog_scan_error', 'Configure Komodo before automatic recovery. RCON or fresh telemetry verifies game status.'); return; setSetting('watchdog_scan_error', 'Register a native host instance before automatic recovery. RCON or fresh telemetry verifies game status.'); return;
} }
const errors: string[] = []; const errors: string[] = [];
const servers = listServers(); const servers = listServers();
+1 -1
View File
@@ -1 +1 @@
export { restartSettling, rolloutWaitReason, refreshRestartRollout, rolloutHealth } from '../provisioning/restarts.js'; export { restartSettling, rolloutWaitReason, rolloutHealth } from '../provisioning/restarts.js';
+2 -2
View File
@@ -100,8 +100,8 @@ function queueStatus(srv: ServerRow, probe?: () => ReturnType<typeof rconForMoni
const server = srv.name; const server = srv.name;
const pending = pendingStatus.get(server); const pending = pendingStatus.get(server);
if (pending) return pending; if (pending) return pending;
// Legacy servers without a recorded host share the configured host's pool. // Host queues keep slow instances from consuming other hosts' slots.
const host = srv.host_id ? `native:${srv.host_id}` : `komodo:${srv.komodo_server || config.komodo.targetServer}`; const host = srv.host_id ? `native:${srv.host_id}` : `unregistered:${srv.name}`;
let queue = statusQueues.get(host); let queue = statusQueues.get(host);
if (!queue) { queue = { active: 0, queued: [] }; statusQueues.set(host, queue); } if (!queue) { queue = { active: 0, queued: [] }; statusQueues.set(host, queue); }
let finish!: (capacity: number | null) => void; let finish!: (capacity: number | null) => void;
+5 -14
View File
@@ -121,7 +121,7 @@ export function getDeployment(id: string, options: { events?: 'all' | 'recent' }
SELECT collection_id, deployment_id FROM server_collection_members SELECT collection_id, deployment_id FROM server_collection_members
UNION ALL SELECT collection_id, deployment_id FROM server_collection_removals) m UNION ALL SELECT collection_id, deployment_id FROM server_collection_removals) m
JOIN server_launch_collections c ON c.id = m.collection_id WHERE m.deployment_id = ? ORDER BY c.created_at, c.id LIMIT 1`).get(id) as { id: string; name: string } | undefined; JOIN server_launch_collections c ON c.id = m.collection_id WHERE m.deployment_id = ? ORDER BY c.created_at, c.id LIMIT 1`).get(id) as { id: string; name: string } | undefined;
return { deployment, backend: native ? 'native' as const : 'komodo' as const, agentOperation: agentOperation ?? null, return { deployment, backend: 'native' as const, agentOperation: agentOperation ?? null,
canCancel: canCancelProvisioningJob(id), canCancelCreation: canCancelCreation(id), cancellation: cancellation ?? null, canCancel: canCancelProvisioningJob(id), canCancelCreation: canCancelCreation(id), cancellation: cancellation ?? null,
workerActivity: workerActivity(), events: deploymentEvents(id, options.events === 'recent'), workerActivity: workerActivity(), events: deploymentEvents(id, options.events === 'recent'),
activity: { ...activitySummary(id), failureExcerpt: latestFailureEvent(id) }, sources, activity: { ...activitySummary(id), failureExcerpt: latestFailureEvent(id) }, sources,
@@ -133,7 +133,7 @@ export function getDeployment(id: string, options: { events?: 'all' | 'recent' }
})(); })();
} }
// This stage is written only after Komodo has confirmed deployment deletion. // This stage is written only after the host agent has confirmed instance removal.
export const REMOVING_SERVER = 'Removing server files and registry'; export const REMOVING_SERVER = 'Removing server files and registry';
export function canCancelRemoval(id: string): boolean { export function canCancelRemoval(id: string): boolean {
@@ -154,15 +154,6 @@ export function cancelRemoval(id: string): boolean {
})(); })();
} }
export function claimRemoval(id: string): boolean {
return db().prepare(`UPDATE server_deployments SET state = 'running'
WHERE id = ? AND action = 'destroy' AND state = 'waiting'
AND (server_deleted_at IS NULL OR (stage = ?
AND NOT EXISTS (SELECT 1 FROM servers s WHERE s.name = server_deployments.server)
AND NOT EXISTS (SELECT 1 FROM server_deployments newer WHERE newer.server = server_deployments.server
AND newer.rowid > server_deployments.rowid)))`).run(id, REMOVING_SERVER).changes > 0;
}
export function canResumeCleanup(id: string): boolean { export function canResumeCleanup(id: string): boolean {
return !!db().prepare(`SELECT d.id FROM server_deployments d return !!db().prepare(`SELECT d.id FROM server_deployments d
WHERE d.id = ? AND (d.server_deleted_at IS NULL OR NOT EXISTS (SELECT 1 FROM servers s WHERE s.name = d.server)) WHERE d.id = ? AND (d.server_deleted_at IS NULL OR NOT EXISTS (SELECT 1 FROM servers s WHERE s.name = d.server))
@@ -177,7 +168,7 @@ export function resumeCleanup(id: string): boolean {
preserveOutcome(id); preserveOutcome(id);
db().prepare("UPDATE server_deployments SET state = 'waiting', finished_at = NULL, error = NULL WHERE id = ?").run(id); db().prepare("UPDATE server_deployments SET state = 'waiting', finished_at = NULL, error = NULL WHERE id = ?").run(id);
recordProgress(id, { phase: 'cleanup', blocker: null, checks: [], observedAt: null, recordProgress(id, { phase: 'cleanup', blocker: null, checks: [], observedAt: null,
event: { kind: 'resumed', message: 'Resumed local cleanup after confirmed Komodo deletion' } }); event: { kind: 'resumed', message: 'Resumed local cleanup after confirmed instance removal' } });
return true; return true;
})(); })();
} }
@@ -207,7 +198,7 @@ export function resumeRebuild(id: string): boolean {
preserveOutcome(id); preserveOutcome(id);
db().prepare("UPDATE server_deployments SET state = 'waiting', finished_at = NULL, error = NULL WHERE id = ?").run(id); db().prepare("UPDATE server_deployments SET state = 'waiting', finished_at = NULL, error = NULL WHERE id = ?").run(id);
recordProgress(id, { phase: 'cleanup', blocker: null, checks: [], observedAt: null, recordProgress(id, { phase: 'cleanup', blocker: null, checks: [], observedAt: null,
event: { kind: 'resumed', message: 'Resumed rebuild cleanup after confirmed Komodo deletion' } }); event: { kind: 'resumed', message: 'Resumed rebuild cleanup after confirmed instance removal' } });
return true; return true;
})(); })();
} }
@@ -218,7 +209,7 @@ export function resumeVerification(id: string): boolean {
if (!canResumeVerification(id)) return false; if (!canResumeVerification(id)) return false;
preserveOutcome(id); preserveOutcome(id);
db().prepare("UPDATE server_deployments SET state = 'awaiting-health', finished_at = NULL, error = NULL WHERE id = ?").run(id); db().prepare("UPDATE server_deployments SET state = 'awaiting-health', finished_at = NULL, error = NULL WHERE id = ?").run(id);
deploymentStage(id, 'Waiting for Komodo health', { phase: 'verify-container', blocker: null, checks: [], observedAt: null, deploymentStage(id, 'Waiting for host agent health', { phase: 'verify-container', blocker: null, checks: [], observedAt: null,
event: { kind: 'resumed', message: 'Resumed verification of the existing server' } }); event: { kind: 'resumed', message: 'Resumed verification of the existing server' } });
return true; return true;
})(); })();
+1 -1
View File
@@ -2,7 +2,7 @@
// Execution is driven exclusively by the provisioning worker. // Execution is driven exclusively by the provisioning worker.
export { requestLifecycle, requestRestart, requestWatchdogLifecycle, escalateWatchdogRestart, provisioningOperation, cancelProvisioning, releaseRestartRequest } from './requests.js'; export { requestLifecycle, requestRestart, requestWatchdogLifecycle, escalateWatchdogRestart, provisioningOperation, cancelProvisioning, releaseRestartRequest } from './requests.js';
export type { LifecycleOperation, RestartSource } from './requests.js'; export type { LifecycleOperation, RestartSource } from './requests.js';
export { queueCreateServer, destroyServer, previewCreation, creationQueue, nextAvailablePort, nextLobbyNumber } from './instances.js'; export { queueCreateServer, destroyServer, previewCreation, creationQueue, nextLobbyNumber } from './instances.js';
export { getDeployment, listDeployments, resumeVerification, resumeCleanup, resumeRebuild, cancelRemoval } from './history.js'; export { getDeployment, listDeployments, resumeVerification, resumeCleanup, resumeRebuild, cancelRemoval } from './history.js';
export { deploymentActivity } from './activity.js'; export { deploymentActivity } from './activity.js';
export { restartSettling, restartRecoveryPending } from './restarts.js'; export { restartSettling, restartRecoveryPending } from './restarts.js';
File diff suppressed because it is too large Load Diff
+3 -3
View File
@@ -45,7 +45,7 @@ export function nativeLaunchSelection(input: Record<string, unknown>, game: stri
if (input.libraryId !== undefined && input.libraryId !== '' && input.libraryId !== libraryId) { if (input.libraryId !== undefined && input.libraryId !== '' && input.libraryId !== libraryId) {
throw new Error(`The current ${game.toUpperCase()} game files changed. Refresh the launch preview.`); throw new Error(`The current ${game.toUpperCase()} game files changed. Refresh the launch preview.`);
} }
const image = input.image === undefined || input.image === '' ? config.komodo.defaultImage : input.image; const image = input.image === undefined || input.image === '' ? config.defaultRuntimeImage : input.image;
if (typeof image !== 'string' || !/^[a-zA-Z0-9][a-zA-Z0-9._/:@-]{0,499}$/.test(image)) throw new Error('Invalid runtime image.'); if (typeof image !== 'string' || !/^[a-zA-Z0-9][a-zA-Z0-9._/:@-]{0,499}$/.test(image)) throw new Error('Invalid runtime image.');
const automatic = input.automatic === true || !input.hostId; const automatic = input.automatic === true || !input.hostId;
if (!automatic) { if (!automatic) {
@@ -673,8 +673,8 @@ async function processJob(job: Job, eligibility: (job: Job) => string | null) {
const recipe = prepared.recipe; const recipe = prepared.recipe;
db().prepare('UPDATE server_launch_recipes SET recipe=CAST(? AS TEXT),recipe_summary=? WHERE deployment_id=?') db().prepare('UPDATE server_launch_recipes SET recipe=CAST(? AS TEXT),recipe_summary=? WHERE deployment_id=?')
.run(Buffer.from(prepared.recipeJson.buffer, prepared.recipeJson.byteOffset, prepared.recipeJson.byteLength), prepared.recipeSummary, job.id); .run(Buffer.from(prepared.recipeJson.buffer, prepared.recipeJson.byteOffset, prepared.recipeJson.byteLength), prepared.recipeSummary, job.id);
if (job.action === 'create') db().prepare(`INSERT INTO servers(name,komodo_type,komodo_name,game,map,game_mode,map_slug,host_dir,config_file,game_id,image,game_port,created_by,container_state) if (job.action === 'create') db().prepare(`INSERT INTO servers(name,game,map,game_mode,map_slug,host_dir,config_file,game_id,image,game_port,created_by,container_state)
VALUES(?,'deployment','',?,?,'zombie',?,?,?,?,?,?,'native','starting')`).run(job.server, recipe.game, recipe.expectedMap, recipe.mapSlug, serverDir(job.server), VALUES(?,?,?,'zombie',?,?,?,?,?,?,'native','starting')`).run(job.server, recipe.game, recipe.expectedMap, recipe.mapSlug, serverDir(job.server),
recipe.configFile, recipe.game === 't6' ? 't6zm' : 't5sp', recipe.image, instance!.port); recipe.configFile, recipe.game === 't6' ? 't6zm' : 't5sp', recipe.image, instance!.port);
for (const file of recipe.files) recordScript(job.server, file.path, file.versionId, file.sha256, file.size, 'native-create'); for (const file of recipe.files) recordScript(job.server, file.path, file.versionId, file.sha256, file.size, 'native-create');
recordProgress(job.id, { checkpoint: { key: 'prepare', label: 'Launch files selected and command payload prepared' } }); recordProgress(job.id, { checkpoint: { key: 'prepare', label: 'Launch files selected and command payload prepared' } });
+1 -1
View File
@@ -37,7 +37,7 @@ export function readProgress(row: { observation_json?: string | null }): Provisi
|| (v.blocker.label !== undefined && !nullableString(v.blocker.label)))) return null; || (v.blocker.label !== undefined && !nullableString(v.blocker.label)))) return null;
if (v.checks.length > 10 || !v.checks.every(c => object(c) && ['container', 'game', 'map', 'plutonium-release', 'plutonium-version', 'occupancy', 'stopped'].includes(String(c.key)) if (v.checks.length > 10 || !v.checks.every(c => object(c) && ['container', 'game', 'map', 'plutonium-release', 'plutonium-version', 'occupancy', 'stopped'].includes(String(c.key))
&& typeof c.label === 'string' && ['pass', 'waiting', 'fail', 'unknown'].includes(String(c.result)) && typeof c.label === 'string' && ['pass', 'waiting', 'fail', 'unknown'].includes(String(c.result))
&& (c.source === null || ['komodo', 'agent', 'rcon', 'telemetry', 'combined', 'manifest'].includes(String(c.source))) && (c.source === null || ['agent', 'rcon', 'telemetry', 'combined', 'manifest'].includes(String(c.source)))
&& typeof c.checkedAt === 'string' && nullableString(c.receivedAt) && scalar(c.expected) && scalar(c.actual) && nullableString(c.message))) return null; && typeof c.checkedAt === 'string' && nullableString(c.receivedAt) && scalar(c.expected) && scalar(c.actual) && nullableString(c.message))) return null;
if (v.checkpoints.length > 100 || !v.checkpoints.every(c => object(c) && typeof c.key === 'string' && typeof c.label === 'string' if (v.checkpoints.length > 100 || !v.checkpoints.every(c => object(c) && typeof c.key === 'string' && typeof c.label === 'string'
&& typeof c.at === 'string' && (c.phase === null || phase(c.phase)))) return null; && typeof c.at === 'string' && (c.phase === null || phase(c.phase)))) return null;
+4 -22
View File
@@ -1,10 +1,8 @@
import { maintenanceAllowed } from '../maintenance/policies.js'; import { maintenanceAllowed } from '../maintenance/policies.js';
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import { komodoConfigured, rconConfigured } from '../config.js';
import { activeBetaRemovalBlocker, getServer } from '../fleet/servers.js'; import { activeBetaRemovalBlocker, getServer } from '../fleet/servers.js';
import { beginDeployment, deploymentStage, finishDeployment, recordDeploymentEvent } from './history.js'; import { beginDeployment, deploymentStage, finishDeployment, recordDeploymentEvent } from './history.js';
import { queueRebuildServer } from './instances.js'; import { queueRebuildServer } from './instances.js';
import { restartRequestInFlight } from './restarts.js';
import { runtimeActions, type RuntimeAction } from '../../provisioning.js'; import { runtimeActions, type RuntimeAction } from '../../provisioning.js';
import { recordProgress } from './progress.js'; import { recordProgress } from './progress.js';
import { nativeInstance, unplacedNativeCreation } from '../backends/native.js'; import { nativeInstance, unplacedNativeCreation } from '../backends/native.js';
@@ -51,22 +49,6 @@ export function requestRestart(server: string, source: RestartSource): Lifecycle
.get(server, row.restarted_at, row.restarted_at) as { id: string } | undefined)?.id; .get(server, row.restarted_at, row.restarted_at) as { id: string } | undefined)?.id;
if (id) db().prepare('UPDATE server_deployments SET submitted_at = COALESCE(submitted_at, ?) WHERE id = ?').run(row.restarted_at, id); if (id) db().prepare('UPDATE server_deployments SET submitted_at = COALESCE(submitted_at, ?) WHERE id = ?').run(row.restarted_at, id);
} }
// Only not-yet-submitted campaign changes can share an activation. A later
// file deployment must never be credited to an already running restart.
// Restorations need their own activation. An older campaign can have
// stale file preconditions after a beta; its failure must not strand
// the rollback that would restore the server.
const rollback = source.kind === 'campaign' && !!db().prepare("SELECT 1 FROM deployments WHERE id = ? AND action = 'rollback'").get(source.id);
if (source.kind === 'campaign' && !row.restarted_at && !rollback && !nativeInstance(server)) {
id = (db().prepare(`SELECT d.id FROM server_deployments d WHERE d.server = ? AND d.server_deleted_at IS NULL AND d.action = 'restart'
AND d.state = 'waiting' AND d.submitted_at IS NULL
AND NOT EXISTS (SELECT 1 FROM native_provisioning n WHERE n.deployment_id=d.id AND n.stage_operation_id IS NOT NULL)
AND EXISTS (SELECT 1 FROM provisioning_sources s WHERE s.operation_id = d.id AND s.kind = 'campaign')
AND NOT EXISTS (SELECT 1 FROM provisioning_sources s WHERE s.operation_id = d.id AND s.kind != 'campaign')
AND NOT EXISTS (SELECT 1 FROM provisioning_sources s JOIN deployments c ON c.id = s.source_id
WHERE s.operation_id = d.id AND s.kind = 'campaign' AND c.action = 'rollback')
ORDER BY d.rowid LIMIT 1`).get(server) as { id: string } | undefined)?.id;
}
if (!id) { if (!id) {
id = beginDeployment(server, 'restart', `${source.kind}:${source.id}`); id = beginDeployment(server, 'restart', `${source.kind}:${source.id}`);
db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id); db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id);
@@ -138,7 +120,7 @@ export function releaseRestartRequest(server: string, source: RestartSource): vo
export function requestLifecycle(server: string, action: RuntimeAction) { export function requestLifecycle(server: string, action: RuntimeAction) {
if (!runtimeActions.includes(action)) return { ok: false as const, status: 400, error: 'Choose start, restart, rebuild, or stop.' }; if (!runtimeActions.includes(action)) return { ok: false as const, status: 400, error: 'Choose start, restart, rebuild, or stop.' };
if (!nativeInstance(server) && !unplacedNativeCreation(server) && (!komodoConfigured() || !rconConfigured())) return { ok: false as const, status: 503, error: 'Configure a host and RCON before provisioning.' }; if (!nativeInstance(server) && !unplacedNativeCreation(server)) return { ok: false as const, status: 503, error: 'Server has no native host instance.' };
return db().transaction(() => { return db().transaction(() => {
const blocker = action === 'rebuild' ? activeBetaRemovalBlocker(server) : null; const blocker = action === 'rebuild' ? activeBetaRemovalBlocker(server) : null;
if (blocker) return { ok: false as const, status: 409, error: blocker }; if (blocker) return { ok: false as const, status: 409, error: blocker };
@@ -187,7 +169,7 @@ function watchdogRequestGuard(server: string, incidentId: string): Extract<Lifec
const row = getServer(server); const row = getServer(server);
if (!row || row.destroyed_at) return { ok: false, status: 404, error: 'Server not found.' }; if (!row || row.destroyed_at) return { ok: false, status: 404, error: 'Server not found.' };
if (row.desired_state === 'stopped') return { ok: false, status: 409, error: 'This server is intentionally stopped.' }; if (row.desired_state === 'stopped') return { ok: false, status: 409, error: 'This server is intentionally stopped.' };
if (!nativeInstance(server) && !komodoConfigured()) return { ok: false, status: 503, error: 'Configure a host before provisioning.' }; if (!nativeInstance(server)) return { ok: false, status: 503, error: 'Configure a host before provisioning.' };
return null; return null;
} }
@@ -237,7 +219,7 @@ export function requestWatchdogLifecycle(server: string, action: 'restart' | 're
JOIN watchdog_incidents i ON i.id = a.incident_id JOIN watchdog_incidents i ON i.id = a.incident_id
WHERE r.server = ? AND r.healthy_at IS NULL AND d.state IN ('failed','interrupted','cancelled') WHERE r.server = ? AND r.healthy_at IS NULL AND d.state IN ('failed','interrupted','cancelled')
AND a.cycle < i.cycle`).get(incidentId, server) as { operation_id: string } | undefined; AND a.cycle < i.cycle`).get(incidentId, server) as { operation_id: string } | undefined;
if (retained && !restartRequestInFlight(server)) { if (retained) {
const transferred = db().prepare('UPDATE restart_rollout SET operation_id = ? WHERE server = ? AND operation_id = ? AND healthy_at IS NULL') const transferred = db().prepare('UPDATE restart_rollout SET operation_id = ? WHERE server = ? AND operation_id = ? AND healthy_at IS NULL')
.run(next.deploymentId, server, retained.operation_id); .run(next.deploymentId, server, retained.operation_id);
if (transferred.changes) { if (transferred.changes) {
@@ -275,7 +257,7 @@ export function escalateWatchdogRestart(incidentId: string, operationId: string,
const slot = db().prepare('SELECT operation_id, requested_at, healthy_at FROM restart_rollout WHERE server = ?').get(operation.server) as const slot = db().prepare('SELECT operation_id, requested_at, healthy_at FROM restart_rollout WHERE server = ?').get(operation.server) as
{ operation_id: string | null; requested_at: string; healthy_at: string | null } | undefined; { operation_id: string | null; requested_at: string; healthy_at: string | null } | undefined;
const anchor = Date.parse(slot?.requested_at ?? operation.submitted_at ?? operation.started_at); const anchor = Date.parse(slot?.requested_at ?? operation.submitted_at ?? operation.started_at);
if (restartRequestInFlight(operation.server) || (slot && (slot.operation_id !== operationId || slot.healthy_at)) if ((slot && (slot.operation_id !== operationId || slot.healthy_at))
|| !Number.isFinite(anchor) || Date.now() - anchor < Math.max(300_000, recoveryTimeoutMs)) { || !Number.isFinite(anchor) || Date.now() - anchor < Math.max(300_000, recoveryTimeoutMs)) {
return { ok: false as const, status: 409, error: 'Waiting for the submitted restart to finish its recovery window.' }; return { ok: false as const, status: 409, error: 'Waiting for the submitted restart to finish its recovery window.' };
} }
+1 -174
View File
@@ -1,20 +1,9 @@
import { fleetSettings } from '../settings.js'; import { fleetSettings } from '../settings.js';
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import { provisioningCancelled } from './cancellation.js';
import { komodoInspectHealthy, komodoRestart, komodoRuntimeAction } from '../integrations/komodo.js';
import { probeServerStatus } from '../fleet/status-evidence.js';
import { deploymentStage, finishDeployment } from './history.js';
import { applyStagedFilesForOperation } from '../fleet/staged-files.js';
import type { RuntimeAction } from '../../provisioning.js'; import type { RuntimeAction } from '../../provisioning.js';
import type { ProvisioningBlocker } from '../../provisioning-detail.js'; import type { ProvisioningBlocker } from '../../provisioning-detail.js';
import { recordProgress, type ProgressUpdate } from './progress.js';
import { nativeDeployment } from '../backends/native.js';
// Komodo queues restart work. Don't mistake the still-running old container
// for a completed restart when an API kick immediately follows the request.
const HEALTH_POLL_MS = 15_000; const HEALTH_POLL_MS = 15_000;
const requestsInFlight = new Map<string, string>();
export const restartRequestInFlight = (server: string) => requestsInFlight.has(server);
interface RestartSlot { interface RestartSlot {
server: string; server: string;
@@ -37,7 +26,7 @@ export function restartRecoveryPending(server: string): boolean {
/** The queued restart may still expose the previous healthy container. */ /** The queued restart may still expose the previous healthy container. */
export function restartSettling(server: string): boolean { export function restartSettling(server: string): boolean {
const slot = slotFor(server); const slot = slotFor(server);
return requestsInFlight.has(server) || !!(slot && !slot.healthy_at return !!(slot && !slot.healthy_at
&& Date.now() - Date.parse(slot.requested_at) < HEALTH_POLL_MS); && Date.now() - Date.parse(slot.requested_at) < HEALTH_POLL_MS);
} }
@@ -46,7 +35,6 @@ export function rolloutWaitReason(server: string, operationId = '', action: Runt
} }
export function rolloutBlocker(server: string, operationId = '', action: RuntimeAction = 'restart'): ProvisioningBlocker | null { export function rolloutBlocker(server: string, operationId = '', action: RuntimeAction = 'restart'): ProvisioningBlocker | null {
if (requestsInFlight.has(server)) return { code: 'operation-conflict', message: 'Waiting for the previous command submission to finish', automatic: true };
const conflict = db().prepare(`SELECT id FROM server_deployments WHERE server = ? AND id != ? AND server_deleted_at IS NULL const conflict = db().prepare(`SELECT id FROM server_deployments WHERE server = ? AND id != ? AND server_deleted_at IS NULL
AND ((action IN ('start','stop','rebuild') AND state IN ('running','awaiting-health')) AND ((action IN ('start','stop','rebuild') AND state IN ('running','awaiting-health'))
OR (action = 'destroy' AND state IN ('waiting','running')))`).get(server, operationId) as { id: string } | undefined; OR (action = 'destroy' AND state IN ('waiting','running')))`).get(server, operationId) as { id: string } | undefined;
@@ -67,167 +55,6 @@ export function rolloutBlocker(server: string, operationId = '', action: Runtime
? { code: 'restart-capacity', message: `Queued to ${action}; waiting for another server to recover`, automatic: true } : null; ? { code: 'restart-capacity', message: `Queued to ${action}; waiting for another server to recover`, automatic: true } : null;
} }
export type RolloutRestartResult =
| { kind: 'started' }
| { kind: 'queued'; message: string }
| { kind: 'failed'; message: string };
/** Reserve capacity before any async I/O; callers still check for players first. */
export async function restartInRollout(server: string, options: { operationId: string; action: 'start' | 'restart' }): Promise<RolloutRestartResult> {
const operationId = options.operationId;
const waitReason = db().transaction(() => {
if (provisioningCancelled(operationId)) return 'Operation was cancelled';
if (!db().prepare('SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL').get(operationId)) return 'Original server was deleted';
const reason = rolloutWaitReason(server, operationId, options.action);
if (reason) return reason;
db().prepare(`INSERT INTO restart_rollout (server, requested_at, operation_id) VALUES (?, ?, ?)
ON CONFLICT(server) DO UPDATE SET requested_at = excluded.requested_at,
checked_at = NULL, healthy_at = NULL, players = NULL, error = NULL, operation_id = excluded.operation_id`)
.run(server, new Date().toISOString(), operationId);
recordProgress(operationId, { blocker: null, recovery: { state: 'held', message: 'Restart recovery capacity reserved.', operationId } });
return null;
}).immediate();
if (waitReason) return { kind: 'queued', message: waitReason };
// Every gate, including rollout capacity, has cleared. No async work may
// intervene between applying these files and submitting the restart.
try {
applyStagedFilesForOperation(operationId, server);
} catch (err) {
const message = `File activation failed: ${(err as Error).message}`;
db().transaction(() => {
db().prepare('DELETE FROM restart_rollout WHERE server = ? AND operation_id = ?').run(server, operationId);
recordProgress(operationId, { recovery: { state: 'released', message: 'Capacity released before any remote command: file activation failed.', operationId } });
finishDeployment(operationId, { ok: false, message });
})();
return { kind: 'failed', message };
}
db().transaction(() => {
db().prepare('UPDATE server_deployments SET submitted_at = ? WHERE id = ?').run(new Date().toISOString(), operationId);
deploymentStage(operationId, options.action === 'start' ? 'Starting container' : 'Restarting container', {
phase: 'container-action', blocker: null, checks: [], checkpoint: { key: 'eligibility', label: 'Eligibility checks passed' },
command: { state: 'attempted', message: `${options.action === 'start' ? 'Start' : 'Restart'} submission attempted.` }
});
})();
requestsInFlight.set(server, operationId);
db().prepare(`UPDATE servers SET container_state = 'restarting', last_error = NULL, rcon_failures = 0, last_reply_snippet = NULL,
game_started_at = NULL, game_ended_at = NULL WHERE name = ?`).run(server);
try {
const result = options.action === 'start' ? await komodoRuntimeAction(server, 'start') : await komodoRestart(server);
if (provisioningCancelled(operationId)) return { kind: 'failed', message: 'Operation was cancelled' };
if (!db().prepare('SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL').get(operationId)) return { kind: 'failed', message: 'Original server was deleted' };
if (result.ok) {
db().transaction(() => {
db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ? AND server_deleted_at IS NULL").run(operationId);
deploymentStage(operationId!, 'Waiting for container and game health (RCON or fresh telemetry)', {
phase: 'verify-container', command: { state: 'accepted', message: 'Komodo accepted the request; recovery still requires verification.' },
checkpoint: { key: 'container-action', label: 'Container command accepted' },
blocker: { code: 'recovery-settling', message: 'Waiting for the recovery settling interval before checking health.', automatic: true }
});
})();
return { kind: 'started' };
}
// These failures cannot have queued a remote restart. Other failures
// (especially a timeout) may have: retain capacity until recovery.
db().transaction(() => {
if (result.kind === 'no-name' || result.kind === 'unconfigured') {
db().prepare('DELETE FROM restart_rollout WHERE server = ? AND operation_id = ?').run(server, operationId);
recordProgress(operationId, { command: { state: 'rejected', message: 'The command could not be submitted: ' + result.message },
recovery: { state: 'released', message: 'Capacity released because no remote command could be submitted.', operationId } });
} else {
db().prepare('UPDATE restart_rollout SET error = ? WHERE server = ? AND operation_id = ?').run(`komodo: ${result.message}`, server, operationId);
recordProgress(operationId, { command: { state: 'uncertain', message: 'The command may have reached Komodo: ' + result.message } });
}
finishDeployment(operationId!, { ok: false, message: `komodo: ${result.message}` });
})();
return { kind: 'failed', message: `komodo: ${result.message}` };
} catch (err) {
if (provisioningCancelled(operationId)) return { kind: 'failed', message: 'Operation was cancelled' };
if (!db().prepare('SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL').get(operationId)) return { kind: 'failed', message: 'Original server was deleted' };
const message = `komodo: ${(err as Error).message}`;
db().transaction(() => {
db().prepare('UPDATE restart_rollout SET error = ? WHERE server = ? AND operation_id = ?').run(message, server, operationId);
recordProgress(operationId, { command: { state: 'uncertain', message: 'The command may have reached Komodo: ' + (err as Error).message } });
finishDeployment(operationId!, { ok: false, message });
})();
return { kind: 'failed', message };
} finally {
// Start the grace period at completion, including slow/failed requests.
db().prepare('UPDATE restart_rollout SET requested_at = ? WHERE server = ? AND operation_id = ?')
.run(new Date().toISOString(), server, operationId);
if (requestsInFlight.get(server) === operationId) requestsInFlight.delete(server);
}
}
let refreshInFlight: Promise<void> | null = null;
/** Poll all occupied slots, even if their originating beta was already closed. */
export function refreshRestartRollout(): Promise<void> {
if (refreshInFlight) return refreshInFlight;
refreshInFlight = refresh().finally(() => { refreshInFlight = null; });
return refreshInFlight;
}
async function refresh(): Promise<void> {
const slots = db().prepare('SELECT * FROM restart_rollout WHERE healthy_at IS NULL ORDER BY requested_at, server')
.all() as RestartSlot[];
for (const slot of slots) {
if (slot.operation_id && nativeDeployment(slot.operation_id)) continue;
if (requestsInFlight.has(slot.server)) continue;
// Recovery capacity transferred to a rebuild remains reserved until
// teardown. Its health is verified by the creation worker, never here.
if (slot.operation_id && db().prepare(`SELECT 1 FROM server_deployments d WHERE d.id = ? AND d.server_deleted_at IS NULL AND (
(d.action = 'rebuild' AND d.state IN ('waiting','queued','running','awaiting-health'))
OR (d.action = 'restart' AND d.state = 'waiting' AND d.submitted_at IS NULL AND EXISTS (
SELECT 1 FROM provisioning_sources s WHERE s.operation_id = d.id AND s.kind = 'watchdog')))`)
.get(slot.operation_id)) continue;
if (Date.now() - Date.parse(slot.checked_at ?? slot.requested_at) < HEALTH_POLL_MS) continue;
let error: string | null = null;
let players: number | null = null;
const checks: NonNullable<ProgressUpdate['checks']> = [];
try {
const health = await komodoInspectHealthy(slot.server);
checks.push({ key: 'container', label: 'Container readiness', result: health.ok ? 'pass' : 'waiting', source: 'komodo', actual: health.state ?? null,
checkedAt: new Date().toISOString(), message: health.ok ? null : health.reason });
if (!health.ok) {
error = `unhealthy: ${health.reason}${health.snippet ? ` snippet=${health.snippet.slice(0, 400)}` : ''}`;
} else {
const status = await probeServerStatus(slot.server, slot.requested_at);
checks.push({ key: 'game', label: 'Game status', result: status.ok ? 'pass' : 'waiting', source: status.ok ? status.source : null,
checkedAt: status.observedAt, receivedAt: status.ok ? status.receivedAt : null, actual: status.ok ? status.players : null,
message: status.ok ? null : status.reason });
if (!status.ok) error = `rcon not answering and no fresh telemetry: ${status.reason}`;
else players = status.players;
}
} catch (err) {
error = `health check: ${(err as Error).message}`;
checks.push({ key: checks.length ? 'game' : 'container', label: checks.length ? 'Game status' : 'Container readiness', result: 'unknown', source: null, message: error });
}
const now = new Date().toISOString();
db().transaction(() => {
const checked = db().prepare(`UPDATE restart_rollout SET checked_at = ?, healthy_at = ?, players = ?, error = ?
WHERE server = ? AND requested_at = ? AND operation_id IS ? AND healthy_at IS NULL`)
.run(now, error ? null : now, players, error, slot.server, slot.requested_at, slot.operation_id);
if (!checked.changes) return;
if (slot.operation_id) recordProgress(slot.operation_id, {
phase: checks.some(check => check.key === 'game') ? 'verify-game' : 'verify-container', checks, observedAt: now,
blocker: error ? { code: 'recovery-health', message: error, automatic: true } : null,
recovery: { state: error ? 'held' : 'released', at: now, operationId: slot.operation_id,
message: error ? 'Restart capacity remains reserved while recovery is checked.' : 'Container and game recovery verified; restart capacity released.' },
checkpoints: error ? [] : [{ key: 'verify-container', label: 'Container readiness verified', at: now }, { key: 'verify-game', label: 'Game status verified', at: now }]
});
if (slot.operation_id && db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'awaiting-health'").get(slot.operation_id)) {
db().prepare('UPDATE server_deployments SET checked_at = ?, players = ? WHERE id = ?').run(now, players, slot.operation_id);
if (error) deploymentStage(slot.operation_id, 'Waiting for health: ' + error);
else {
deploymentStage(slot.operation_id, 'Verified container and game health (RCON or fresh telemetry)');
finishDeployment(slot.operation_id, { ok: true });
}
}
})();
}
}
export function rolloutHealth(server: string): { ok: true; players: number } | { ok: false; reason: string } { export function rolloutHealth(server: string): { ok: true; players: number } | { ok: false; reason: string } {
const slot = slotFor(server); const slot = slotFor(server);
if (slot?.healthy_at) return { ok: true, players: slot.players ?? 0 }; if (slot?.healthy_at) return { ok: true, players: slot.players ?? 0 };
+5 -170
View File
@@ -1,24 +1,11 @@
import { maintenanceAllowed } from '../maintenance/policies.js'; import { maintenanceAllowed } from '../maintenance/policies.js';
import { resolveRecoveredHealthOperation } from '../maintenance/recovery.js';
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import { provisioningCancelled } from './cancellation.js';
import { komodoConfigured, rconConfigured } from '../config.js';
import { getServer } from '../fleet/servers.js'; import { getServer } from '../fleet/servers.js';
import { deploymentStage, finishDeployment, REMOVING_SERVER } from './history.js'; import { rolloutBlocker } from './restarts.js';
import { processCreations, processDestructions, rebuildServer } from './instances.js';
import { komodoInspectHealthy, komodoRuntimeAction } from '../integrations/komodo.js';
import { latestGameSubmission, probePlayerEvidence } from '../fleet/status-evidence.js';
import { refreshRestartRollout, restartInRollout, rolloutBlocker } from './restarts.js';
import { cancelProvisioning } from './requests.js'; import { cancelProvisioning } from './requests.js';
import type { RuntimeAction } from '../../provisioning.js'; import type { RuntimeAction } from '../../provisioning.js';
import type { ProvisioningBlocker } from '../../provisioning-detail.js'; import type { ProvisioningBlocker } from '../../provisioning-detail.js';
import { recordProgress } from './progress.js';
import { nativeInstance, nativeDeployment } from '../backends/native.js';
import { processNativeProvisioning } from './native.js'; import { processNativeProvisioning } from './native.js';
import { trackActivity } from '../pipeline/activity.js';
const stoppedStates = new Set(['stopped', 'exited', 'created', 'dead']);
export const isStopped = (state?: string) => stoppedStates.has(state?.toLowerCase() ?? '');
interface Source { kind: 'campaign' | 'beta' | 'maintenance' | 'watchdog'; source_id: string } interface Source { kind: 'campaign' | 'beta' | 'maintenance' | 'watchdog'; source_id: string }
const sourcesFor = (id: string) => db().prepare('SELECT kind, source_id FROM provisioning_sources WHERE operation_id = ?').all(id) as Source[]; const sourcesFor = (id: string) => db().prepare('SELECT kind, source_id FROM provisioning_sources WHERE operation_id = ?').all(id) as Source[];
@@ -92,20 +79,13 @@ function blocker(server: string, id: string, action: RuntimeAction): Provisionin
} }
interface Operation { id: string; server: string; action: RuntimeAction; state: string; stage: string } interface Operation { id: string; server: string; action: RuntimeAction; state: string; stage: string }
let ticking: Promise<void> | null = null;
let awaitedPass: Promise<void> | null = null; let awaitedPass: Promise<void> | null = null;
export function processProvisioning(options: { wait?: boolean } = {}): Promise<void> { export function processProvisioning(options: { wait?: boolean } = {}): Promise<void> {
if (options.wait !== false && awaitedPass) return awaitedPass; if (options.wait !== false && awaitedPass) return awaitedPass;
// Native jobs own their individual flights. A slow observation must not // Each job owns its flight, so slow observations do not block new work.
// prevent the next scheduler pass admitting newly arrived native work. const pass = nativePass(options);
const native = nativePass(options); if (options.wait === false) return pass;
if (!ticking) ticking = trackActivity('legacy-provisioning', 'Processing legacy provisioning', run) awaitedPass = pass.finally(() => { awaitedPass = null; });
.finally(() => { ticking = null; });
if (options.wait === false) {
void ticking.catch(error => console.error('[worker] legacy provisioning error:', error));
return native;
}
awaitedPass = Promise.all([native, ticking]).then(() => {}).finally(() => { awaitedPass = null; });
return awaitedPass; return awaitedPass;
} }
@@ -126,148 +106,3 @@ function nativePass(options: { wait?: boolean }): Promise<void> {
return blocker(operation.server, operation.id, typed.action)?.message ?? null; return blocker(operation.server, operation.id, typed.action)?.message ?? null;
}, options); }, options);
} }
async function run(): Promise<void> {
await processCreations();
await processDestructions();
await refreshRestartRollout();
const operations = db().prepare(`SELECT id, server, action, state, stage FROM server_deployments
WHERE server_deleted_at IS NULL AND action IN ('start','restart','stop','rebuild') AND (state = 'waiting' OR (action = 'stop' AND state = 'awaiting-health'))
ORDER BY started_at, rowid`).all() as Operation[];
for (const operation of operations) {
if (nativeDeployment(operation.id)) continue;
try { await processOperation(operation); }
catch (err) {
if (provisioningCancelled(operation.id)) continue;
// Observation failures are retried; remote mutations are never replayed.
deploymentStage(operation.id, 'Waiting for verification: ' + (err as Error).message, {
checks: [], blocker: { code: 'observation-unavailable', message: (err as Error).message, automatic: true }
});
}
}
}
async function processOperation(operation: Operation): Promise<void> {
const { id, server, action } = operation;
if (provisioningCancelled(id)) return;
// An explicit cleanup resume follows confirmed remote deletion. Complete
// that durable checkpoint even if the incident now needs attention or the
// Watchdog is paused; never strand the reserved replacement identity.
if (action === 'rebuild' && operation.stage === REMOVING_SERVER) { await rebuildServer(id); return; }
if (operation.state === 'waiting') {
const heldBy: Source[] = [];
const eligibility = sourceEligibility(operation, source => heldBy.push(source));
if (eligibility === 'cancelled') { cancelProvisioning(id, 'Requester cancelled or replaced this operation'); return; }
if (eligibility === 'held') {
const source = heldBy[0];
const href = source ? source.kind === 'campaign' ? `/campaigns/${encodeURIComponent(source.source_id)}`
: source.kind === 'beta' ? `/betas/${encodeURIComponent(source.source_id)}`
: source.kind === 'watchdog' ? `/maintenance/${encodeURIComponent(source.source_id)}`
: `/maintenance?view=activity#job-${encodeURIComponent(source.source_id)}` : null;
deploymentStage(id, 'Waiting for the requester to resume', { phase: 'eligibility', checks: [],
blocker: { code: 'requester-held', message: 'Waiting for the requester to resume', automatic: false, href, label: 'Requesting work' } });
return;
}
}
if (!getServer(server)) { finishDeployment(id, { ok: false, message: 'Server is no longer registered.' }); return; }
const watchdog = sourcesFor(id).some(source => source.kind === 'watchdog');
if (!komodoConfigured() || (!rconConfigured() && !watchdog)) { deploymentStage(id, 'Waiting for Komodo and RCON configuration', {
phase: 'eligibility', checks: [], blocker: { code: 'configuration', message: 'Waiting for Komodo and RCON configuration', automatic: false, href: '/settings', label: 'Integration settings' }
}); return; }
if (operation.state === 'awaiting-health') {
const health = await komodoInspectHealthy(server);
if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'awaiting-health'").get(id)) return;
recordProgress(id, { phase: 'verify-stopped', checks: [{ key: 'stopped', label: 'Container stopped', source: 'komodo',
result: isStopped(health.state) ? 'pass' : health.state ? 'waiting' : 'unknown', actual: health.state ?? null,
message: health.ok ? null : health.reason }], blocker: isStopped(health.state) ? null
: { code: 'stopped-state', message: 'Waiting for an explicit stopped container state.', automatic: true } });
if (isStopped(health.state)) {
db().transaction(() => {
db().prepare("UPDATE servers SET container_state = ?, last_players = NULL, last_roster = NULL, game_started_at = NULL, game_ended_at = NULL WHERE name = ?").run(health.state, server);
deploymentStage(id, 'Verified container stopped', { checkpoint: { key: 'verify-stopped', label: 'Stopped container state verified' } });
finishDeployment(id, { ok: true });
})();
} else deploymentStage(id, 'Waiting for container to stop' + (!health.ok ? ': ' + health.reason : ''));
return;
}
const reason = blocker(server, id, action);
if (reason) { deploymentStage(id, reason.message, { phase: 'eligibility', checks: [], blocker: reason }); return; }
const healthRecovery = watchdog && !!db().prepare(`SELECT 1 FROM provisioning_sources s
JOIN watchdog_incidents i ON i.id = s.source_id JOIN server_deployments d ON d.id = s.operation_id
WHERE s.operation_id = ? AND s.kind = 'watchdog' AND i.kind = 'health' AND d.triggered_by LIKE 'watchdog:%'`).get(id);
const epoch = latestGameSubmission(server);
const health = await komodoInspectHealthy(server);
if (provisioningCancelled(id)) return;
if (healthRecovery && epoch !== latestGameSubmission(server)) return;
if (healthRecovery && !health.ok && !['stopped','exited','created','dead','unhealthy','starting','restarting'].includes(health.state?.toLowerCase() ?? '')) {
deploymentStage(id, 'Waiting for current container health before automatic recovery', { phase: 'eligibility', checks: [],
blocker: { code: 'observation-unavailable', message: health.reason, automatic: true } });
return;
}
if (action !== 'start' && !isStopped(health.state)) {
const evidence = await probePlayerEvidence(server);
if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'waiting'").get(id)) return;
const players = evidence.players;
if (healthRecovery && epoch !== latestGameSubmission(server)) return;
if (healthRecovery && health.ok && players !== null) {
// The fresh occupancy check also proves that the game recovered.
// Empty is not a reason to execute an obsolete health restart.
resolveRecoveredHealthOperation(id, players, evidence.source);
return;
}
recordProgress(id, { phase: 'eligibility', checks: [{ key: 'occupancy', label: 'Player occupancy', result: players === 0 ? 'pass' : players === null ? 'unknown' : 'waiting',
source: evidence.source, checkedAt: evidence.observedAt, receivedAt: evidence.receivedAt, expected: 0, actual: players }],
blocker: players === 0 ? null : { code: players === null ? 'players-unknown' : 'players-present',
message: players === null ? 'Waiting for a fresh player count' : `Waiting for ${players} player(s) to leave`, automatic: true } });
db().prepare('UPDATE server_deployments SET players = ?, checked_at = ? WHERE id = ?').run(players, new Date().toISOString(), id);
if (players !== 0) {
deploymentStage(id, players === null ? 'Waiting for a fresh player count' : `Waiting for ${players} player(s) to leave`);
return;
}
} else if (action !== 'start') {
if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'waiting'").get(id)) return;
recordProgress(id, { phase: 'eligibility', blocker: null, checks: [{ key: 'stopped', label: 'Stopped container permits disruption',
source: 'komodo', result: 'pass', actual: health.state ?? null }] });
}
// Requests can arrive during the probes. Recheck before claiming the action.
const latestReason = blocker(server, id, action);
if (latestReason) { deploymentStage(id, latestReason.message, { phase: 'eligibility', blocker: latestReason }); return; }
if (sourceEligibility(operation) !== 'ready') return;
if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'waiting'").get(id)) return;
if (action === 'rebuild') { await rebuildServer(id); return; }
const claim = db().prepare("UPDATE server_deployments SET state = 'running' WHERE id = ? AND server_deleted_at IS NULL AND state = 'waiting'").run(id);
if (!claim.changes) return;
if (action === 'start' || action === 'restart') {
const result = await restartInRollout(server, { operationId: id, action });
if (provisioningCancelled(id)) return;
if (result.kind === 'queued') {
db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id);
deploymentStage(id, result.message);
}
return;
}
db().transaction(() => {
db().prepare('UPDATE server_deployments SET submitted_at = ? WHERE id = ?').run(new Date().toISOString(), id);
deploymentStage(id, 'Stopping container', { phase: 'container-action', blocker: null, checks: [],
checkpoint: { key: 'eligibility', label: 'Eligibility checks passed' }, command: { state: 'attempted', message: 'Stop submission attempted.' } });
})();
// Persist intent first; a lost reply must not issue another stop later.
try {
const result = await komodoRuntimeAction(server, 'stop');
if (provisioningCancelled(id)) return;
if (!result.ok) {
recordProgress(id, { command: { state: result.kind === 'no-name' || result.kind === 'unconfigured' ? 'rejected' : 'uncertain', message: result.message } });
finishDeployment(id, { ok: false, message: result.message }); return;
}
db().transaction(() => {
db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ?").run(id);
deploymentStage(id, 'Waiting for container to stop', { phase: 'verify-stopped',
checkpoint: { key: 'container-action', label: 'Stop command accepted' },
command: { state: 'accepted', message: 'Komodo accepted the stop request; stopped state still requires verification.' } });
})();
} catch (err) {
if (provisioningCancelled(id)) return;
recordProgress(id, { command: { state: 'uncertain', message: (err as Error).message } });
finishDeployment(id, { ok: false, message: (err as Error).message });
}
}
@@ -9,7 +9,7 @@
const progress = $derived(detail.progress); const progress = $derived(detail.progress);
let expanded = $state<Record<string, boolean>>({}); let expanded = $state<Record<string, boolean>>({});
const evidenceLabels = { pass: 'Passed', waiting: 'Waiting', fail: 'Failed', unknown: 'Unknown' }; const evidenceLabels = { pass: 'Passed', waiting: 'Waiting', fail: 'Failed', unknown: 'Unknown' };
const sources = { komodo: 'Komodo', agent: 'Host agent', rcon: 'RCON', telemetry: 'Telemetry', combined: 'RCON and telemetry', manifest: 'Plutonium manifest' }; const sources = { agent: 'Host agent', rcon: 'RCON', telemetry: 'Telemetry', combined: 'RCON and telemetry', manifest: 'Plutonium manifest' };
</script> </script>
<section class="execution" aria-labelledby="execution-title"> <section class="execution" aria-labelledby="execution-title">
-27
View File
@@ -1,27 +0,0 @@
export type KomodoPortFailure =
| 'unconfigured'
| 'no-name'
| 'unreachable'
| 'http-error'
| 'unparseable'
| 'no-udp-in-response';
/** Short fix hint per failure reason, shown under the Komodo probe result. */
export function komodoFixHint(reason: KomodoPortFailure, httpStatus?: number): string {
switch (reason) {
case 'unconfigured':
return 'Set KOMODO_ADDRESS, KOMODO_API_KEY, and KOMODO_API_SECRET in the container env.';
case 'no-name':
return 'This server row has no komodo_name — set it via SQL or via the fleet config.';
case 'unreachable':
return 'Network, DNS, or timeout. Check that KOMODO_ADDRESS is reachable from this container.';
case 'http-error':
if (httpStatus === 401 || httpStatus === 403) return 'Auth rejected — verify KOMODO_API_KEY and KOMODO_API_SECRET.';
if (httpStatus === 404) return 'Endpoint or deployment not found — check komodo_name matches the Komodo UI exactly.';
return `Komodo returned ${httpStatus ?? 'an error'}. Check the endpoint path and credentials.`;
case 'unparseable':
return 'Response was not JSON — is KOMODO_ADDRESS pointing at the Core API (not the Periphery)?';
case 'no-udp-in-response':
return 'Parsed the response but no UDP port matched our known shapes. See the snippet below and extend extractUdpPort.';
}
}
+15 -24
View File
@@ -8,35 +8,27 @@ export interface ProgressGroup { key: string; label: string; steps: ProgressStep
const phaseLabels: Record<ProvisioningPhase, string> = { const phaseLabels: Record<ProvisioningPhase, string> = {
requested: 'Request recorded', eligibility: 'Check eligibility and capacity', 'startup-queue': 'Wait for a startup slot', requested: 'Request recorded', eligibility: 'Check eligibility and capacity', 'startup-queue': 'Wait for a startup slot',
prepare: 'Prepare server files', 'create-deployment': 'Create Komodo deployment', 'container-action': 'Start container', prepare: 'Prepare server files', 'container-action': 'Start container',
'remove-container': 'Stop and remove container', 'delete-deployment': 'Delete Komodo deployment', cleanup: 'Remove files and registration', cleanup: 'Remove files and registration',
'verify-container': 'Confirm container readiness', 'verify-game': 'Verify game status', 'verify-map': 'Verify selected engine map', 'verify-container': 'Confirm container readiness', 'verify-game': 'Verify game status', 'verify-map': 'Verify selected engine map',
'verify-plutonium': 'Verify Plutonium revision', 'verify-plutonium': 'Verify Plutonium revision',
'verify-stopped': 'Confirm container is stopped' 'verify-stopped': 'Confirm container is stopped'
}; };
type GroupDefinition = { key: string; label: string; phases: ProvisioningPhase[] }; type GroupDefinition = { key: string; label: string; phases: ProvisioningPhase[] };
const request: GroupDefinition = { key: 'request', label: 'Request and eligibility', phases: ['requested', 'eligibility'] }; const request: GroupDefinition = { key: 'request', label: 'Request and eligibility', phases: ['requested', 'eligibility'] };
const creation: GroupDefinition = { key: 'create', label: 'Create and start', phases: ['startup-queue', 'prepare', 'create-deployment', 'container-action'] };
const removal: GroupDefinition = { key: 'remove', label: 'Remove the instance', phases: ['remove-container', 'delete-deployment', 'cleanup'] };
const readiness: GroupDefinition = { key: 'verify', label: 'Verify readiness', phases: ['verify-container', 'verify-game', 'verify-map', 'verify-plutonium'] }; const readiness: GroupDefinition = { key: 'verify', label: 'Verify readiness', phases: ['verify-container', 'verify-game', 'verify-map', 'verify-plutonium'] };
function definitions(action: ProvisioningAction, native = false): GroupDefinition[] { function definitions(action: ProvisioningAction): GroupDefinition[] {
if (native) { const command: GroupDefinition = { key: 'command', label: `${actionLabels[action]} on host agent`, phases: ['container-action'] };
const command: GroupDefinition = { key: 'command', label: `${actionLabels[action]} on host agent`, phases: ['container-action'] }; if (action === 'create' || action === 'rebuild') return [request,
if (action === 'create' || action === 'rebuild') return [request, { key: 'create', label: action === 'rebuild' ? 'Rebuild on host agent' : 'Create and start on host agent', phases: ['startup-queue', 'prepare', 'container-action'] }, readiness];
{ key: 'create', label: action === 'rebuild' ? 'Rebuild on host agent' : 'Create and start on host agent', phases: ['startup-queue', 'prepare', 'container-action'] }, readiness]; if (action === 'destroy') return [request, command, { key: 'cleanup', label: 'Retire instance', phases: ['cleanup'] }];
if (action === 'destroy') return [request, command, { key: 'cleanup', label: 'Retire instance', phases: ['cleanup'] }]; return [request, command, { ...readiness, phases: action === 'stop' ? ['verify-stopped'] : ['verify-container', 'verify-game'] }];
}
if (action === 'create') return [{ ...request, phases: ['requested'] }, creation, readiness];
if (action === 'rebuild') return [request, removal, { ...creation, label: 'Create a fresh instance' }, readiness];
if (action === 'destroy') return [request, removal];
return [request, { key: 'command', label: `${actionLabels[action]} the existing container`, phases: ['container-action'] },
{ ...readiness, phases: action === 'stop' ? ['verify-stopped'] : ['verify-container', 'verify-game'] }];
} }
export function provisioningGroups(detail: ProvisioningDetail): ProgressGroup[] { export function provisioningGroups(detail: ProvisioningDetail): ProgressGroup[] {
const progress = detail.progress; const progress = detail.progress;
if (!progress) return []; if (!progress) return [];
const groups = definitions(detail.deployment.action, detail.backend === 'native'); const groups = definitions(detail.deployment.action);
const phases = groups.flatMap(group => group.phases); const phases = groups.flatMap(group => group.phases);
const current = progress.phase ? phases.indexOf(progress.phase) : -1; const current = progress.phase ? phases.indexOf(progress.phase) : -1;
const active = activeProvisioningStates.includes(detail.deployment.state); const active = activeProvisioningStates.includes(detail.deployment.state);
@@ -54,11 +46,10 @@ export function provisioningGroups(detail: ProvisioningDetail): ProgressGroup[]
: isCurrent && active ? progress.blocker || ['waiting', 'queued'].includes(detail.deployment.state) ? 'waiting' : 'current' : isCurrent && active ? progress.blocker || ['waiting', 'queued'].includes(detail.deployment.state) ? 'waiting' : 'current'
: current >= 0 && index >= current && detail.deployment.state === 'cancelled' ? 'cancelled' : current >= 0 && index >= current && detail.deployment.state === 'cancelled' ? 'cancelled'
: active && current >= 0 && index > current ? 'pending' : 'unknown'; : active && current >= 0 && index > current ? 'pending' : 'unknown';
const label = detail.backend === 'native' && key === 'startup-queue' ? 'Admit to startup capacity' const label = key === 'startup-queue' ? 'Admit to startup capacity'
: detail.backend === 'native' && key === 'prepare' ? 'Prepare launch payload' : key === 'prepare' ? 'Prepare launch payload'
: detail.backend === 'native' && key === 'container-action' ? `${actionLabels[detail.deployment.action]} instance through host agent` : key === 'container-action' ? `${actionLabels[detail.deployment.action]} instance through host agent`
: key === 'container-action' && ['start', 'restart', 'stop'].includes(detail.deployment.action) : phaseLabels[key];
? `${actionLabels[detail.deployment.action]} existing container` : phaseLabels[key];
return { key, label, state, at: check?.checkedAt ?? checkpoint?.at ?? (isCurrent ? progress.phaseStartedAt : null) }; return { key, label, state, at: check?.checkedAt ?? checkpoint?.at ?? (isCurrent ? progress.phaseStartedAt : null) };
}); });
const states = steps.map(step => step.state); const states = steps.map(step => step.state);
@@ -168,7 +159,7 @@ export function isProvisioningDetail(value: unknown, id: string): value is Provi
if (value.cancellation !== undefined && value.cancellation !== null && !(record(value.cancellation) if (value.cancellation !== undefined && value.cancellation !== null && !(record(value.cancellation)
&& text(value.cancellation.cleanupId) && text(value.cancellation.state) && text(value.cancellation.stage) && text(value.cancellation.cleanupId) && text(value.cancellation.state) && text(value.cancellation.stage)
&& nullableText(value.cancellation.error))) return false; && nullableText(value.cancellation.error))) return false;
if (value.backend !== undefined && !['native', 'komodo'].includes(value.backend as string)) return false; if (value.backend !== undefined && !['native'].includes(value.backend as string)) return false;
if (value.workerActivity !== undefined && (!Array.isArray(value.workerActivity) || !value.workerActivity.every(item => if (value.workerActivity !== undefined && (!Array.isArray(value.workerActivity) || !value.workerActivity.every(item =>
record(item) && text(item.key) && text(item.label) && timestamp(item.startedAt)))) return false; record(item) && text(item.key) && text(item.label) && timestamp(item.startedAt)))) return false;
if (value.agentOperation !== undefined && value.agentOperation !== null && !(record(value.agentOperation) if (value.agentOperation !== undefined && value.agentOperation !== null && !(record(value.agentOperation)
@@ -212,7 +203,7 @@ export function isProvisioningDetail(value: unknown, id: string): value is Provi
if (!Array.isArray(progress.checks) || !progress.checks.every(check => record(check) if (!Array.isArray(progress.checks) || !progress.checks.every(check => record(check)
&& ['container', 'game', 'map', 'plutonium-release', 'plutonium-version', 'occupancy', 'stopped'].includes(check.key as string) && ['container', 'game', 'map', 'plutonium-release', 'plutonium-version', 'occupancy', 'stopped'].includes(check.key as string)
&& ['pass', 'waiting', 'fail', 'unknown'].includes(check.result as string) && ['pass', 'waiting', 'fail', 'unknown'].includes(check.result as string)
&& [null, 'komodo', 'agent', 'rcon', 'telemetry', 'combined', 'manifest'].includes(check.source as null | string) && [null, 'agent', 'rcon', 'telemetry', 'combined', 'manifest'].includes(check.source as null | string)
&& text(check.label) && timestamp(check.checkedAt) && nullableTime(check.receivedAt) && text(check.label) && timestamp(check.checkedAt) && nullableTime(check.receivedAt)
&& scalar(check.expected) && scalar(check.actual) && nullableText(check.message))) return false; && scalar(check.expected) && scalar(check.actual) && nullableText(check.message))) return false;
if (!Array.isArray(progress.checkpoints) || !progress.checkpoints.every(checkpoint => record(checkpoint) if (!Array.isArray(progress.checkpoints) || !progress.checkpoints.every(checkpoint => record(checkpoint)
+1 -1
View File
@@ -30,7 +30,7 @@ export function rconFixHint(reason: RconFailure): string {
case 'no-config': case 'no-config':
return 'Set RCON_PASSWORD in the container env and restart.'; return 'Set RCON_PASSWORD in the container env and restart.';
case 'no-port': case 'no-port':
return 'Set game_port on the server row, or configure Komodo so it can resolve the port.'; return 'Check the host agent and the game port assigned to this server.';
case 'send-error': case 'send-error':
return 'DNS or route problem — check the RCON host is reachable from this container.'; return 'DNS or route problem — check the RCON host is reachable from this container.';
case 'timeout': case 'timeout':
+1 -4
View File
@@ -13,7 +13,7 @@ export function createFleetStore(serverName?: string) {
}); });
const fleetError = writable(''); const fleetError = writable('');
const fleetRefreshing = writable(false); const fleetRefreshing = writable(false);
const fleetIntegrations = writable<{ rcon: boolean; komodo: boolean; native?: boolean; hostsReady?: number; hostsTotal?: number; rconTimeoutMs: number; defaultRconHost: string } | null>(null); const fleetIntegrations = writable<{ rcon: boolean; native?: boolean; hostsReady?: number; hostsTotal?: number; rconTimeoutMs: number; defaultRconHost: string } | null>(null);
const observed = new Map<string, { server: FleetServer; status: unknown; match: unknown; report: unknown; phase: string; value: FleetServer }>(); const observed = new Map<string, { server: FleetServer; status: unknown; match: unknown; report: unknown; phase: string; value: FleetServer }>();
const freshnessPhase = derived([fleet, statusByServer, telemetryByServer, uiNow], ([snapshot, live, reports, now]) => { const freshnessPhase = derived([fleet, statusByServer, telemetryByServer, uiNow], ([snapshot, live, reports, now]) => {
@@ -63,7 +63,6 @@ export function createFleetStore(serverName?: string) {
return next; return next;
}); });
let snapshotServerNow = 0; let snapshotServerNow = 0;
function hydrateFleet(snapshot: { servers: FleetServer[]; operations: FleetOperation[]; integrations: NonNullable<ReturnType<typeof getIntegrations>>; serverNow: number }) { function hydrateFleet(snapshot: { servers: FleetServer[]; operations: FleetOperation[]; integrations: NonNullable<ReturnType<typeof getIntegrations>>; serverNow: number }) {
if (!browser || snapshot.serverNow < snapshotServerNow) return; if (!browser || snapshot.serverNow < snapshotServerNow) return;
@@ -74,8 +73,6 @@ export function createFleetStore(serverName?: string) {
} }
function getIntegrations() { return get(fleetIntegrations); } function getIntegrations() { return get(fleetIntegrations); }
let inFlight: Promise<void> | null = null; let inFlight: Promise<void> | null = null;
let refreshRequested = false; let refreshRequested = false;
function refreshFleet(afterCurrent = false): Promise<void> { function refreshFleet(afterCurrent = false): Promise<void> {
-2
View File
@@ -165,8 +165,6 @@
</div>{/if} </div>{/if}
{#if integrations?.native && !integrations.hostsReady} {#if integrations?.native && !integrations.hostsReady}
<div class="setup-note"><span><strong>Prepare a game host</strong> · {integrations.hostsTotal ? 'Hosts are connected or enrolled but need setup before new deployments.' : 'Add a host, prepare its runtime, and download game files.'}</span><a href="/hosts">Review hosts →</a></div> <div class="setup-note"><span><strong>Prepare a game host</strong> · {integrations.hostsTotal ? 'Hosts are connected or enrolled but need setup before new deployments.' : 'Add a host, prepare its runtime, and download game files.'}</span><a href="/hosts">Review hosts →</a></div>
{:else if $page.data.user?.role === 'admin' && $fleetIntegrations && !integrations.native && (!integrations.komodo || !integrations.rcon)}
<div class="setup-note"><span><strong>Finish system setup</strong> · {!integrations.komodo ? 'Komodo' : ''}{!integrations.komodo && !integrations.rcon ? ' and ' : ''}{!integrations.rcon ? 'RCON' : ''} not configured.</span><a href="/settings">Review setup →</a></div>
{/if} {/if}
{#if problems.length || attentionOperations.length || starting || provisioning} {#if problems.length || attentionOperations.length || starting || provisioning}
+1 -1
View File
@@ -4,7 +4,7 @@ import { hostApiError, installerCommand, objectBody, requireManagementOrigin } f
import { presentHostPreparation } from '$lib/server/hosts/preparation.js'; import { presentHostPreparation } from '$lib/server/hosts/preparation.js';
import { config } from '$lib/server/config.js'; import { config } from '$lib/server/config.js';
export function GET({ request }: { request: Request }) { requireManagementOrigin(request); return json({ hosts: listHosts().map(presentHostPreparation), defaultRuntimeImage: config.komodo.defaultImage }); } export function GET({ request }: { request: Request }) { requireManagementOrigin(request); return json({ hosts: listHosts().map(presentHostPreparation), defaultRuntimeImage: config.defaultRuntimeImage }); }
export async function POST({ request }: { request: Request }) { export async function POST({ request }: { request: Request }) {
requireManagementOrigin(request); requireManagementOrigin(request);
try { try {
+1 -1
View File
@@ -10,7 +10,7 @@ import { kickProvisioning } from '$lib/server/pipeline/worker.js';
type Event = { request: Request; params: { id: string } }; type Event = { request: Request; params: { id: string } };
export function GET({ request, params }: Event) { export function GET({ request, params }: Event) {
requireManagementOrigin(request); const host = getHost(params.id); if (!host) error(404, 'Host not found.'); requireManagementOrigin(request); const host = getHost(params.id); if (!host) error(404, 'Host not found.');
return json({ host: presentHostPreparation(host), instances: hostInstances(host.id), serverNow: Date.now(), calibration: capacityCalibration(host.id), defaultRuntimeImage: config.komodo.defaultImage, deletion: hostDeletionStatus(host.id) }); return json({ host: presentHostPreparation(host), instances: hostInstances(host.id), serverNow: Date.now(), calibration: capacityCalibration(host.id), defaultRuntimeImage: config.defaultRuntimeImage, deletion: hostDeletionStatus(host.id) });
} }
export async function PATCH({ request, params }: Event) { export async function PATCH({ request, params }: Event) {
requireManagementOrigin(request); requireManagementOrigin(request);
@@ -1,77 +0,0 @@
import { json } from '@sveltejs/kit';
import type { RequestHandler } from './$types';
import { getServer } from '$lib/server/fleet/servers.js';
import { config } from '$lib/server/config.js';
import { tcpProbeAll } from '$lib/server/integrations/netProbe.js';
/**
* Multi-target TCP reachability probe. Fires parallel connect attempts at a
* fixed matrix of candidate hosts + the configured KOMODO_ADDRESS host, so
* we can see at a glance which network paths reach anything and where
* packets die (DNS, routing, or refused-at-port).
*/
export const POST: RequestHandler = async ({ params }) => {
const name = params.name;
if (!name) return json({ ok: false, error: 'missing server name' }, { status: 400 });
const row = getServer(name);
if (!row) return json({ ok: false, error: `unknown server: ${name}` }, { status: 404 });
// Parse the configured Komodo URL to extract host+port.
let komodoHost: string | null = null;
let komodoPort: number | null = null;
if (config.komodo.address) {
try {
const u = new URL(config.komodo.address);
komodoHost = u.hostname;
komodoPort = u.port ? Number.parseInt(u.port, 10) : u.protocol === 'https:' ? 443 : 80;
} catch {
/* leave null */
}
}
const probePort = komodoPort ?? 9120;
// Build the trial matrix. Every trial hits the same target port (the
// Komodo port) via a different network path, so we can see which paths
// reach the same service.
const trials: Array<{ label: string; host: string; port: number }> = [
{ label: `127.0.0.1:${probePort} (container loopback)`, host: '127.0.0.1', port: probePort },
{ label: `host.docker.internal:${probePort} (bridge gateway alias)`, host: 'host.docker.internal', port: probePort },
{ label: `172.17.0.1:${probePort} (default bridge gateway)`, host: '172.17.0.1', port: probePort }
];
if (komodoHost && !trials.some((t) => t.host === komodoHost)) {
trials.push({
label: `${komodoHost}:${probePort} (KOMODO_ADDRESS)`,
host: komodoHost,
port: probePort
});
}
// RCON's default host is worth testing too — even though RCON is UDP,
// if TCP to the same host also fails, it's a routing/firewall issue,
// not a UDP-specific one.
const rconHost = row.rcon_host ?? config.rcon.host;
if (rconHost && !trials.some((t) => t.host === rconHost)) {
trials.push({
label: `${rconHost}:${probePort} (RCON_HOST)`,
host: rconHost,
port: probePort
});
}
const results = await tcpProbeAll(trials, config.httpTimeoutMs);
return json({
ok: true,
config: {
komodoAddress: config.komodo.address || null,
komodoHost,
komodoPort,
rconHost,
probeTimeoutMs: config.httpTimeoutMs
},
results
});
};
@@ -1,27 +0,0 @@
import { json } from '@sveltejs/kit';
import type { RequestHandler } from './$types';
import { komodoResolvePortDetailed } from '$lib/server/integrations/komodo.js';
import { getServer, setServerField } from '$lib/server/fleet/servers.js';
/**
* Trigger a fresh Komodo GetDeployment lookup for a specific server and
* return the structured result — including a snippet of the response so
* the UI can render "here's where we found the port" (or "here's what
* Komodo returned; extend extractUdpPort for this shape").
*
* On success, the resolved port is written to servers.game_port so the
* next RCON attempt uses it without waiting for another sweep tick.
*/
export const POST: RequestHandler = async ({ params }) => {
const name = params.name;
if (!name) return json({ ok: false, reason: 'no-name', detail: 'missing server name' }, { status: 400 });
const row = getServer(name);
if (!row) return json({ ok: false, reason: 'no-name', detail: `unknown server: ${name}` }, { status: 404 });
const result = await komodoResolvePortDetailed(name);
if (result.ok) {
setServerField(name, 'game_port', result.port);
}
return json(result);
};
+13 -42
View File
@@ -1,59 +1,30 @@
import { json } from '@sveltejs/kit'; import { json } from '@sveltejs/kit';
import { config, komodoConfigured, rconConfigured } from '$lib/server/config.js';
import { listServers } from '$lib/server/fleet/servers.js'; import { listServers } from '$lib/server/fleet/servers.js';
import { listServerKeys } from '$lib/server/fleet/server-keys.js'; import { listServerKeys } from '$lib/server/fleet/server-keys.js';
import { serverObservation } from '$lib/server/fleet/status.js'; import { serverObservation } from '$lib/server/fleet/status.js';
import { freshStatusTimestamp } from '$lib/server-freshness.js'; import { freshStatusTimestamp } from '$lib/server-freshness.js';
import { tcpProbe } from '$lib/server/integrations/netProbe.js';
import type { SetupCheck, SystemStatus } from '$lib/system-status.js'; import type { SetupCheck, SystemStatus } from '$lib/system-status.js';
import { listHosts } from '$lib/server/hosts/index.js'; import { listHosts } from '$lib/server/hosts/index.js';
import { listLibraries } from '$lib/server/game-files/index.js'; import { listLibraries } from '$lib/server/game-files/index.js';
export async function GET({ url }: { url: URL }) { export async function GET() {
const servers = listServers(); const servers = listServers();
const readyServers = servers.filter(server => serverObservation(server).health === 'online').length; const readyServers = servers.filter(server => serverObservation(server).health === 'online').length;
const rconServers = servers.filter(server => !server.last_error && freshStatusTimestamp(server.last_seen_at, Date.now()) && server.container_state === 'running').length; const rconServers = servers.filter(server => !server.last_error && freshStatusTimestamp(server.last_seen_at, Date.now()) && server.container_state === 'running').length;
const savedKeys = listServerKeys().length; const savedKeys = listServerKeys().length;
const hosts = listHosts(); const hosts = listHosts();
if (hosts.length || !komodoConfigured()) { const readyHosts = hosts.filter(host => host.readiness === 'ready');
const readyHosts = hosts.filter(host => host.readiness === 'ready'); const connected = hosts.filter(host => host.connected);
const connected = hosts.filter(host => host.connected); const versions = listLibraries().flatMap(game => game.versions.filter(version => version.id === game.latestVersionId));
const versions = listLibraries().flatMap(game => game.versions.filter(version => version.id === game.latestVersionId)); const cached = versions.filter(version => version.caches.some(cache => cache.status === 'ready' && readyHosts.some(host => host.id === cache.hostId)));
const cached = versions.filter(version => version.caches.some(cache => cache.status === 'ready' && readyHosts.some(host => host.id === cache.hostId))); const runtimes = connected.filter(host => host.inventory.runtimeImages?.length);
const runtimes = connected.filter(host => host.inventory.runtimeImages?.length);
const checks: SetupCheck[] = [
{ id: 'hosts', title: 'Host agents', state: readyHosts.length ? 'ready' : hosts.length ? 'attention' : 'unconfigured', detail: `${connected.length} connected, ${readyHosts.length} deployment-ready, ${hosts.length} enrolled or awaiting enrollment.`, href: '/hosts', action: hosts.length ? 'Review hosts' : 'Add host' },
{ id: 'runtime', title: 'Plutainer runtime', state: runtimes.length ? 'ready' : 'attention', detail: runtimes.length ? `${runtimes.length} connected hosts report cached runtime images.` : 'Hosts prepare the configured Plutainer runtime automatically.', href: '/hosts', action: 'View host setup' },
{ id: 'game-files', title: 'T5 / T6 game files', state: cached.length ? 'ready' : 'attention', detail: `${versions.length} games have uploaded files; ${cached.length} have their latest files verified on a ready host.`, href: '/game-files', action: 'Manage game files' },
{ id: 'rcon', title: 'Agent-local game connection', state: rconServers ? 'ready' : 'unverified', detail: rconServers ? `${rconServers} servers have fresh RCON responses.` : 'The host agent checks RCON locally after deployment. Controller access to game ports is not required.', href: servers.length ? '/' : '/hosts', action: servers.length ? 'Review fleet' : 'Review hosts' },
{ id: 'keys', title: 'Saved server keys', state: savedKeys ? 'ready' : 'attention', detail: `${savedKeys} saved ${savedKeys === 1 ? 'key' : 'keys'}. Validity is checked with Plutonium at startup.`, href: '/server-keys', action: 'Manage keys' },
{ id: 'first-server', title: 'First server ready', state: readyServers ? 'ready' : 'attention', detail: readyServers ? `${readyServers} of ${servers.length} servers are responding.` : servers.length ? 'Registered servers are awaiting fresh healthy reports. Inspect their host and diagnostics.' : 'Choose a host to launch your first server with the latest game files.', href: servers.length ? '/' : '/provisioning', action: servers.length ? 'Review fleet' : 'Create a server' }
];
return json({ checkedAt: new Date().toISOString(), checks, readyServers, totalServers: servers.length, savedKeys } satisfies SystemStatus, { headers: { 'cache-control': 'no-store' } });
}
const missing = [!config.komodo.address && 'KOMODO_ADDRESS', !config.komodo.apiKey && 'KOMODO_API_KEY', !config.komodo.apiSecret && 'KOMODO_API_SECRET'].filter(Boolean);
const connection: SetupCheck = { id: 'komodo', title: 'Komodo connection', state: komodoConfigured() ? 'unverified' : 'unconfigured', detail: komodoConfigured() ? 'Credentials configured. Run connection check to verify host reachability.' : `Set ${missing.join(', ')} in the app environment.`, href: '#setup-guide', action: 'Setup instructions' };
if (komodoConfigured() && url.searchParams.get('probe') === '1') {
try {
const address = new URL(config.komodo.address);
if (!['http:', 'https:'].includes(address.protocol)) throw new Error('invalid address');
const result = await Promise.race([
tcpProbe('Komodo', address.hostname, Number(address.port || (address.protocol === 'https:' ? 443 : 80)), 3000),
new Promise<null>(resolve => setTimeout(() => resolve(null), 4000))
]);
const reachable = result?.outcome === 'connected';
connection.state = reachable ? 'ready' : 'attention';
connection.detail = reachable ? 'Host reachable. This network check does not validate API credentials; deployment previews check launch prerequisites.' : 'Host unreachable from Electric Cherry. Check the address, DNS, port, and network access.';
} catch { connection.state = 'attention'; connection.detail = 'Komodo address is invalid or could not be reached. Check KOMODO_ADDRESS.'; }
}
const checks: SetupCheck[] = [ const checks: SetupCheck[] = [
connection, { id: 'hosts', title: 'Host agents', state: readyHosts.length ? 'ready' : hosts.length ? 'attention' : 'unconfigured', detail: `${connected.length} connected, ${readyHosts.length} deployment-ready, ${hosts.length} enrolled or awaiting enrollment.`, href: '/hosts', action: hosts.length ? 'Review hosts' : 'Add host' },
{ id: 'target', title: 'Deployment host', state: config.komodo.targetServer ? 'ready' : 'unconfigured', detail: config.komodo.targetServer ? `New servers target ${config.komodo.targetServer}.` : 'Set KOMODO_TARGET_SERVER to the exact host name shown in Komodo.', href: '#setup-guide', action: 'Setup instructions' }, { id: 'runtime', title: 'Plutainer runtime', state: runtimes.length ? 'ready' : 'attention', detail: runtimes.length ? `${runtimes.length} connected hosts report cached runtime images.` : 'Hosts prepare the configured Plutainer runtime automatically.', href: '/hosts', action: 'View host setup' },
{ id: 'rcon', title: 'Game server connection', state: !rconConfigured() ? 'unconfigured' : rconServers ? 'ready' : 'unverified', detail: !rconConfigured() ? 'Set RCON_PASSWORD to the password used by your game servers.' : rconServers ? `${rconServers} servers have fresh successful RCON reports.` : 'RCON password configured. A running server is needed to verify a response.', href: servers.length ? '/' : '#setup-guide', action: servers.length ? 'Review fleet' : 'Setup instructions' }, { id: 'game-files', title: 'T5 / T6 game files', state: cached.length ? 'ready' : 'attention', detail: `${versions.length} games have uploaded files; ${cached.length} have their latest files verified on a ready host.`, href: '/game-files', action: 'Manage game files' },
{ id: 'game-files', title: 'Game files on the host', state: 'unverified', detail: 'T5 and T6 game-file volumes must be populated on the Komodo host. Their contents cannot be verified from this app.', href: '#game-files', action: 'Game-file instructions' }, { id: 'rcon', title: 'Agent-local game connection', state: rconServers ? 'ready' : 'unverified', detail: rconServers ? `${rconServers} servers have fresh RCON responses.` : 'The host agent checks RCON locally after deployment. Controller access to game ports is not required.', href: servers.length ? '/' : '/hosts', action: servers.length ? 'Review fleet' : 'Review hosts' },
{ id: 'keys', title: 'Saved server keys', state: savedKeys ? 'ready' : 'attention', detail: `${savedKeys} saved ${savedKeys === 1 ? 'key' : 'keys'}. Key presence does not confirm validity with Plutonium.`, href: '/server-keys', action: savedKeys ? 'Manage keys' : 'Add a key' }, { id: 'keys', title: 'Saved server keys', state: savedKeys ? 'ready' : 'attention', detail: `${savedKeys} saved ${savedKeys === 1 ? 'key' : 'keys'}. Validity is checked with Plutonium at startup.`, href: '/server-keys', action: 'Manage keys' },
{ id: 'first-server', title: 'First server ready', state: readyServers ? 'ready' : 'attention', detail: readyServers ? `${readyServers} of ${servers.length} servers are responding.` : servers.length ? 'Servers are registered but none has a fresh healthy report. Open diagnostics to investigate.' : 'Create your first server after completing the setup checks.', href: servers.length ? '/' : '/provisioning', action: servers.length ? 'Review fleet' : 'Create a server' } { id: 'first-server', title: 'First server ready', state: readyServers ? 'ready' : 'attention', detail: readyServers ? `${readyServers} of ${servers.length} servers are responding.` : servers.length ? 'Registered servers are awaiting fresh healthy reports. Inspect their host and diagnostics.' : 'Choose a host to launch your first server with the latest game files.', href: servers.length ? '/' : '/provisioning', action: servers.length ? 'Review fleet' : 'Create a server' }
]; ];
const body: SystemStatus = { checkedAt: new Date().toISOString(), checks, readyServers, totalServers: servers.length, savedKeys }; return json({ checkedAt: new Date().toISOString(), checks, readyServers, totalServers: servers.length, savedKeys } satisfies SystemStatus, { headers: { 'cache-control': 'no-store' } });
return json(body, { headers: { 'cache-control': 'no-store' } });
} }
+6 -222
View File
@@ -29,7 +29,6 @@
import ChatPanel from '$ui/components/ChatPanel.svelte'; import ChatPanel from '$ui/components/ChatPanel.svelte';
import { gametypeLabel } from '$ui/gametype.js'; import { gametypeLabel } from '$ui/gametype.js';
import { decodeRconError, rconFixHint } from '$ui/rconError.js'; import { decodeRconError, rconFixHint } from '$ui/rconError.js';
import { komodoFixHint, type KomodoPortFailure } from '$ui/komodoError.js';
import Card from '$ui/components/primitives/Card.svelte'; import Card from '$ui/components/primitives/Card.svelte';
import Badge from '$ui/components/primitives/Badge.svelte'; import Badge from '$ui/components/primitives/Badge.svelte';
@@ -41,29 +40,6 @@
import Breadcrumbs from '$ui/components/primitives/Breadcrumbs.svelte'; import Breadcrumbs from '$ui/components/primitives/Breadcrumbs.svelte';
import EmptyState from '$ui/components/primitives/EmptyState.svelte'; import EmptyState from '$ui/components/primitives/EmptyState.svelte';
interface KomodoProbeResult {
ok: boolean;
port?: number;
reason?: KomodoPortFailure;
detail?: string;
httpStatus?: number;
snippet?: string;
url?: string;
}
type NetOutcome = 'connected' | 'refused' | 'timeout' | 'dns-fail' | 'unreachable' | 'error';
interface NetTrial {
label: string; host: string; port: number;
resolvedIp: string | null; outcome: NetOutcome; elapsedMs: number; detail?: string;
}
interface NetDiagnosis {
ok: boolean;
config?: {
komodoAddress: string | null; komodoHost: string | null; komodoPort: number | null;
rconHost: string; probeTimeoutMs: number;
};
results?: NetTrial[];
error?: string;
}
interface RconProbeResult { interface RconProbeResult {
ok: boolean; ok: boolean;
endpoint?: { host: string; port: number } | null; endpoint?: { host: string; port: number } | null;
@@ -82,7 +58,6 @@
native_instance_id?: string | null; host_id?: string | null; instance_generation?: number | null; game_library_id?: string | null; native_instance_id?: string | null; host_id?: string | null; instance_generation?: number | null; game_library_id?: string | null;
host?: import('$lib/hosts.js').Host | null; host?: import('$lib/hosts.js').Host | null;
network?: { bindAddress: string; localPort: number; publicAddress: string | null; publicPort: number | null } | null; network?: { bindAddress: string; localPort: number; publicAddress: string | null; publicPort: number | null } | null;
komodo_type: string; komodo_name: string; komodo_server: string | null;
game_port: number | null; rcon_host: string | null; game_port: number | null; rcon_host: string | null;
game: string | null; map: string | null; game: string | null; map: string | null;
last_players: number | null; last_checked_at: string | null; last_players: number | null; last_checked_at: string | null;
@@ -98,9 +73,6 @@
relPath: string; kind?: string; scriptVersionId: number | null; relPath: string; kind?: string; scriptVersionId: number | null;
sha256: string; size: number; updatedAt: string; source: string | null; sha256: string; size: number; updatedAt: string; source: string | null;
} }
interface Integrations {
rcon: boolean; komodo: boolean; rconTimeoutMs: number; defaultRconHost: string;
}
interface DeploymentRow { interface DeploymentRow {
id: string; action: string; started_at: string; finished_at: string | null; id: string; action: string; started_at: string; finished_at: string | null;
servers: Array<{ server: string; restart_state: string; players: number | null; error: string | null; files_changed: number }>; servers: Array<{ server: string; restart_state: string; players: number | null; error: string | null; files_changed: number }>;
@@ -108,7 +80,7 @@
let name = $derived($page.params.name ?? ''); let name = $derived($page.params.name ?? '');
const detail = $derived(createFleetStore(name)); const detail = $derived(createFleetStore(name));
const { fleet, fleetServers, fleetError, fleetIntegrations, refreshFleet } = $derived(detail); const { fleet, fleetServers, fleetError, refreshFleet } = $derived(detail);
$effect(() => { $effect(() => {
detail.hydrateFleet(data.serverSnapshot); detail.hydrateFleet(data.serverSnapshot);
return detail.startFleetMonitor(); return detail.startFleetMonitor();
@@ -117,13 +89,8 @@
const visibleFleet = $derived($fleet.receivedAt ? $fleetServers : data.serverSnapshot.servers); const visibleFleet = $derived($fleet.receivedAt ? $fleetServers : data.serverSnapshot.servers);
let server = $derived((visibleFleet.find(s => s.name === name) as Server | undefined) ?? null); let server = $derived((visibleFleet.find(s => s.name === name) as Server | undefined) ?? null);
const native = $derived(!!server?.native_instance_id); const native = $derived(!!server?.native_instance_id);
let integrations = $derived($fleetIntegrations ?? data.serverSnapshot.integrations);
let history = $state<DeploymentRow[]>([]); let history = $state<DeploymentRow[]>([]);
let inventory = $state<InventoryEntry[]>([]); let inventory = $state<InventoryEntry[]>([]);
let komodoProbe = $state<KomodoProbeResult | null>(null);
let probing = $state(false);
let netDiagnosis = $state<NetDiagnosis | null>(null);
let diagnosing = $state(false);
let rconProbe = $state<RconProbeResult | null>(null); let rconProbe = $state<RconProbeResult | null>(null);
let probingRcon = $state(false); let probingRcon = $state(false);
let destroyOpen = $state(false); let destroyOpen = $state(false);
@@ -171,7 +138,7 @@
tabLoading = tab === 'history' || tab === 'inventory'; tabLoading = tab === 'history' || tab === 'inventory';
return pollVisible(signal => loadTab(currentName, tab, signal), 10000); return pollVisible(signal => loadTab(currentName, tab, signal), 10000);
}); });
$effect(() => { name; komodoProbe = null; rconProbe = null; netDiagnosis = null; }); $effect(() => { name; rconProbe = null; });
const fresh = $derived(server ? freshServer(server, $uiNow) : false); const fresh = $derived(server ? freshServer(server, $uiNow) : false);
let effectivePlayers = $derived(fresh ? server?.last_players ?? null : null); let effectivePlayers = $derived(fresh ? server?.last_players ?? null : null);
@@ -214,29 +181,6 @@
return `${ep.host}:${ep.port}`; return `${ep.host}:${ep.port}`;
} }
async function probeKomodo(): Promise<void> {
if (probing || !name) return;
probing = true; komodoProbe = null;
try {
const res = await fetch(`/api/servers/${encodeURIComponent(name)}/probe-komodo`, { method: 'POST' });
komodoProbe = (await res.json()) as KomodoProbeResult;
if (komodoProbe.ok && server) {
await refreshFleet();
}
} catch (err) {
komodoProbe = { ok: false, reason: 'unreachable', detail: (err as Error).message };
} finally { probing = false; }
}
async function diagnoseNetwork(): Promise<void> {
if (diagnosing || !name) return;
diagnosing = true; netDiagnosis = null;
try {
const res = await fetch(`/api/servers/${encodeURIComponent(name)}/diagnose-network`, { method: 'POST' });
netDiagnosis = (await res.json()) as NetDiagnosis;
} catch (err) {
netDiagnosis = { ok: false, error: (err as Error).message };
} finally { diagnosing = false; }
}
async function probeRcon(): Promise<void> { async function probeRcon(): Promise<void> {
if (probingRcon || !name) return; if (probingRcon || !name) return;
probingRcon = true; rconProbe = null; probingRcon = true; rconProbe = null;
@@ -337,8 +281,8 @@
{#if $fleetError}<div class="ui-error" role="alert">{$fleetError}<Button size="sm" onclick={() => refreshFleet(true)}>Retry</Button></div>{/if} {#if $fleetError}<div class="ui-error" role="alert">{$fleetError}<Button size="sm" onclick={() => refreshFleet(true)}>Retry</Button></div>{/if}
{#if hasDiagnosticIssue} {#if hasDiagnosticIssue}
<section class="connection-issue" aria-label="Server connection issue"> <section class="connection-issue" aria-label="Server connection issue">
<div><strong>{effectiveHealth === 'stale' ? 'Server report is outdated' : 'Server is not responding'}</strong><p>Player count is unknown. {effectiveSeen ? `Last successful report ${ago(effectiveSeen, $uiNow)}.` : 'No successful report has been received.'} {native && !server.host?.connected ? 'The host agent is offline; the game may still be running.' : !native && integrations && !integrations.rcon ? 'RCON is not configured.' : ''}</p></div> <div><strong>{effectiveHealth === 'stale' ? 'Server report is outdated' : 'Server is not responding'}</strong><p>Player count is unknown. {effectiveSeen ? `Last successful report ${ago(effectiveSeen, $uiNow)}.` : 'No successful report has been received.'} {native && !server.host?.connected ? 'The host agent is offline; the game may still be running.' : ''}</p></div>
<div class="issue-actions"><Button size="sm" onclick={() => changeTab('diagnostics')}>Diagnose connection</Button><Button size="sm" variant="ghost" onclick={() => changeTab('logs')}>Open logs</Button>{#if native && server.host_id}<Button size="sm" href={`/hosts/${server.host_id}`}>Check host</Button>{:else if integrations && (!integrations.rcon || !integrations.komodo)}<Button size="sm" href="/settings">Check system setup</Button>{/if}</div> <div class="issue-actions"><Button size="sm" onclick={() => changeTab('diagnostics')}>Diagnose connection</Button><Button size="sm" variant="ghost" onclick={() => changeTab('logs')}>Open logs</Button>{#if native && server.host_id}<Button size="sm" href={`/hosts/${server.host_id}`}>Check host</Button>{:else}<Button size="sm" href="/settings">Check system setup</Button>{/if}</div>
</section> </section>
{/if} {/if}
<div class="tabs-row"> <div class="tabs-row">
@@ -410,13 +354,6 @@
<p class="sub">Current player count: {server.status_source === 'telemetry' ? 'game telemetry' : server.status_source === 'rcon' ? 'RCON' : 'unavailable'}. RCON probes below show that connection's separate results.</p> <p class="sub">Current player count: {server.status_source === 'telemetry' ? 'game telemetry' : server.status_source === 'rcon' ? 'RCON' : 'unavailable'}. RCON probes below show that connection's separate results.</p>
{#if native} {#if native}
<div class="diag-grid"><div class="kv"><span class="k">Game connection</span><span class="v">RCON runs locally through the host agent</span></div><div class="kv"><span class="k">Host agent</span><span class="v {server.host?.connected ? 'ok' : 'bad'}">{server.host?.connectionState ?? 'unknown'}</span></div><div class="kv"><span class="k">Last heartbeat</span><span class="v">{server.host?.lastHeartbeat ? ago(server.host.lastHeartbeat, $uiNow) : 'Not received'}</span></div></div> <div class="diag-grid"><div class="kv"><span class="k">Game connection</span><span class="v">RCON runs locally through the host agent</span></div><div class="kv"><span class="k">Host agent</span><span class="v {server.host?.connected ? 'ok' : 'bad'}">{server.host?.connectionState ?? 'unknown'}</span></div><div class="kv"><span class="k">Last heartbeat</span><span class="v">{server.host?.lastHeartbeat ? ago(server.host.lastHeartbeat, $uiNow) : 'Not received'}</span></div></div>
{:else if integrations}
<div class="diag-grid">
<div class="kv"><span class="k">Endpoint</span><span class="v mono">{endpointLabel(effectiveEndpoint)}</span></div>
<div class="kv"><span class="k">RCON password</span><span class="v mono {integrations.rcon ? 'ok' : 'bad'}">{integrations.rcon ? 'configured' : 'missing'}</span></div>
<div class="kv"><span class="k">Komodo</span><span class="v mono {integrations.komodo ? 'ok' : 'bad'}">{integrations.komodo ? 'configured' : 'missing'}</span></div>
<div class="kv"><span class="k">Timeout</span><span class="v mono">{integrations.rconTimeoutMs} ms</span></div>
</div>
{/if} {/if}
{#if effectiveReason} {#if effectiveReason}
@@ -445,14 +382,7 @@
{native ? 'Probe RCON through agent' : 'Probe RCON'} {native ? 'Probe RCON through agent' : 'Probe RCON'}
</Button> </Button>
{/if} {/if}
{#if !native && !effectiveEndpoint && integrations?.komodo} {#if server.host_id}<Button variant="secondary" size="sm" href={`/hosts/${server.host_id}`}>Inspect host</Button>{/if}
<Button variant="secondary" size="sm" onclick={probeKomodo} disabled={probing} loading={probing}>
Probe Komodo
</Button>
{/if}
{#if !native}<Button variant="secondary" size="sm" onclick={diagnoseNetwork} disabled={diagnosing} loading={diagnosing}>
Diagnose network
</Button>{:else if server.host_id}<Button variant="secondary" size="sm" href={`/hosts/${server.host_id}`}>Inspect host</Button>{/if}
</div> </div>
{#if rconProbe?.ok} {#if rconProbe?.ok}
@@ -466,91 +396,12 @@
</div> </div>
{/if} {/if}
{#if !native && netDiagnosis}
{#if netDiagnosis.ok && netDiagnosis.results}
<!-- svelte-ignore a11y_no_noninteractive_tabindex (Keyboard users need to scroll the table.) -->
<div class="table-scroll" role="region" aria-label="Network diagnostics" tabindex="0">
<table class="net-table">
<thead>
<tr>
<th>Trial</th><th>Resolved</th><th>Outcome</th>
<th class="num">Elapsed</th><th>Detail</th>
</tr>
</thead>
<tbody>
{#each netDiagnosis.results as r (r.label)}
<tr>
<td class="mono">{r.label}</td>
<td class="mono">{r.resolvedIp ?? '—'}</td>
<td>
<Badge
variant={r.outcome === 'connected' ? 'online' :
r.outcome === 'refused' ? 'warning' :
'danger'}
uppercase
>{r.outcome}</Badge>
</td>
<td class="mono num">{r.elapsedMs} ms</td>
<td class="mono detail-cell">{r.detail ?? ''}</td>
</tr>
{/each}
</tbody>
</table>
</div>
{#if netDiagnosis.config}
<div class="net-meta">
KOMODO_ADDRESS = <code>{netDiagnosis.config.komodoAddress ?? '(unset)'}</code>
&nbsp;·&nbsp; RCON_HOST = <code>{netDiagnosis.config.rconHost}</code>
&nbsp;·&nbsp; probe timeout = <code>{netDiagnosis.config.probeTimeoutMs} ms</code>
</div>
{/if}
{:else}
<div class="alert alert-danger">
<Badge variant="danger" uppercase>Error</Badge>
<span class="alert-detail">{netDiagnosis.error ?? 'network diagnostic failed'}</span>
</div>
{/if}
{/if}
{#if !native && komodoProbe}
{#if komodoProbe.ok}
<div class="alert alert-ok">
<Badge variant="online" uppercase>Resolved</Badge>
<span class="alert-detail">
Port <strong>{komodoProbe.port}</strong> cached — RCON will retry on the next tick.
</span>
</div>
{#if komodoProbe.snippet}
<details class="snippet-details">
<summary>Show response snippet</summary>
<pre class="snippet">{komodoProbe.snippet}</pre>
</details>
{/if}
{:else}
<div class="alert alert-danger">
<Badge variant="danger" uppercase>{komodoProbe.reason}</Badge>
<span class="alert-detail">
{komodoProbe.detail}{#if komodoProbe.httpStatus} · HTTP {komodoProbe.httpStatus}{/if}
</span>
</div>
{#if komodoProbe.url}
<div class="url-row"><span class="k">Tried</span><code>{komodoProbe.url}</code></div>
{/if}
{#if komodoProbe.reason}
<div class="hint">{komodoFixHint(komodoProbe.reason, komodoProbe.httpStatus)}</div>
{/if}
{#if komodoProbe.snippet}
<pre class="snippet">{komodoProbe.snippet}</pre>
{/if}
{/if}
{/if}
<div class="panel-head divider"> <div class="panel-head divider">
<h2>Configuration</h2> <h2>Configuration</h2>
</div> </div>
<div class="diag-grid"> <div class="diag-grid">
{#if native}<div class="kv"><span class="k">Host</span><span class="v"><a href={`/hosts/${server.host_id}`}>{server.host?.name ?? server.host_id}</a></span></div><div class="kv"><span class="k">Instance</span><span class="v mono">{server.native_instance_id}</span></div><div class="kv"><span class="k">Generation</span><span class="v mono">{server.instance_generation}</span></div><div class="kv"><span class="k">Game-file version</span><span class="v mono"><a href="/game-files">{server.game_library_id}</a></span></div>{#if server.network}<div class="kv"><span class="k">Local UDP binding</span><span class="v mono">{server.network.bindAddress}:{server.network.localPort}</span></div><div class="kv"><span class="k">Public / NAT endpoint</span><span class="v mono">{server.network.publicAddress || server.host?.publicAddress || server.host?.hostname || 'Not configured'}:{server.network.publicPort ?? server.network.localPort}</span></div>{/if} {#if native}<div class="kv"><span class="k">Host</span><span class="v"><a href={`/hosts/${server.host_id}`}>{server.host?.name ?? server.host_id}</a></span></div><div class="kv"><span class="k">Instance</span><span class="v mono">{server.native_instance_id}</span></div><div class="kv"><span class="k">Generation</span><span class="v mono">{server.instance_generation}</span></div><div class="kv"><span class="k">Game-file version</span><span class="v mono"><a href="/game-files">{server.game_library_id}</a></span></div>{#if server.network}<div class="kv"><span class="k">Local UDP binding</span><span class="v mono">{server.network.bindAddress}:{server.network.localPort}</span></div><div class="kv"><span class="k">Public / NAT endpoint</span><span class="v mono">{server.network.publicAddress || server.host?.publicAddress || server.host?.hostname || 'Not configured'}:{server.network.publicPort ?? server.network.localPort}</span></div>{/if}
{:else}<div class="kv"><span class="k">Komodo</span><span class="v mono">{server.komodo_type} · {server.komodo_name}</span></div><div class="kv"><span class="k">RCON override</span><span class="v mono">{server.rcon_host ?? '(default)'}</span></div>{/if} {/if}
<div class="kv"><span class="k">Game port</span><span class="v mono">{server.game_port ?? '(unresolved)'}</span></div> <div class="kv"><span class="k">Game port</span><span class="v mono">{server.game_port ?? '(unresolved)'}</span></div>
<div class="kv"><span class="k">Game</span><span class="v mono">{server.game ?? '(unset)'}</span></div> <div class="kv"><span class="k">Game</span><span class="v mono">{server.game ?? '(unset)'}</span></div>
<div class="kv"><span class="k">Map</span><span class="v mono">{server.map ?? '(unset)'}</span></div> <div class="kv"><span class="k">Map</span><span class="v mono">{server.map ?? '(unset)'}</span></div>
@@ -777,19 +628,6 @@
margin-top: var(--space-4); margin-top: var(--space-4);
} }
.url-row {
display: flex;
gap: var(--space-2);
margin-top: var(--space-2);
padding: var(--space-2) var(--space-3);
background: var(--surface-sunken);
border-radius: var(--radius-md);
font-family: var(--font-mono);
font-size: var(--text-xs);
}
.url-row .k { color: var(--text-tertiary); }
.url-row code { color: var(--text-primary); word-break: break-all; }
/* Snippet details */ /* Snippet details */
.snippet-details { .snippet-details {
margin-top: var(--space-3); margin-top: var(--space-3);
@@ -818,60 +656,6 @@
word-break: break-all; word-break: break-all;
} }
/* Network table */
.table-scroll { max-width: 100%; overflow-x: auto; }
.table-scroll:focus-visible { box-shadow: var(--focus-ring); }
.net-table {
min-width: 600px;
width: 100%;
margin-top: var(--space-4);
border-collapse: separate;
border-spacing: 0;
background: var(--surface-sunken);
border: 1px solid var(--border-subtle);
border-radius: var(--radius-md);
overflow: hidden;
font-size: var(--text-sm);
}
.net-table thead th {
text-align: left;
padding: var(--space-3) var(--space-4);
font-family: var(--font-body);
font-size: var(--text-xs);
font-weight: var(--weight-semibold);
text-transform: uppercase;
letter-spacing: var(--tracking-label);
color: var(--text-tertiary);
border-bottom: 1px solid var(--border-subtle);
}
.net-table tbody td {
padding: var(--space-3) var(--space-4);
border-bottom: 1px solid var(--border-subtle);
color: var(--text-primary);
vertical-align: top;
}
.net-table tbody tr:last-child td { border-bottom: none; }
.net-table .num { text-align: right; }
.net-table .detail-cell {
color: var(--text-tertiary);
font-size: var(--text-xs);
word-break: break-word;
max-width: 320px;
}
.net-meta {
margin-top: var(--space-3);
font-family: var(--font-mono);
font-size: var(--text-xs);
color: var(--text-tertiary);
}
.net-meta code {
color: var(--text-primary);
background: var(--surface-sunken);
padding: 2px 6px;
border-radius: var(--radius-sm);
}
/* Inventory */ /* Inventory */
.inv-list { .inv-list {
display: flex; display: flex;
+23
View File
@@ -0,0 +1,23 @@
import * as fs from 'node:fs/promises';
import { join } from 'node:path';
import { vi } from 'vitest';
/** Read-only agent transport backed by fixture files. Mutations still fail. */
export async function mockAgentFiles() {
const backend = await import('../../src/lib/server/backends/native.js');
const { serverDir } = await import('../../src/lib/server/config.js');
return vi.spyOn(backend, 'nativeRequest').mockImplementation(async (instance, kind, payload = {}) => {
const path = join(serverDir(instance.server), String(payload.path ?? '.'));
let result: unknown;
if (kind === 'files.read') {
const content = await fs.readFile(path).catch((error: NodeJS.ErrnoException) => { if (error.code === 'ENOENT') return null; throw error; });
result = { contentBase64: content?.toString('base64') ?? null };
} else if (kind === 'files.stat') {
const stat = await fs.stat(path).catch((error: NodeJS.ErrnoException) => { if (error.code === 'ENOENT') return null; throw error; });
result = stat ? { exists: true, size: stat.size, mtime: stat.mtime.toISOString(), isDirectory: stat.isDirectory(), isFile: stat.isFile() } : null;
} else if (kind === 'files.list') {
result = { entries: (await fs.readdir(path, { withFileTypes: true })).map(entry => ({ name: entry.name, isDirectory: entry.isDirectory(), isFile: entry.isFile() })) };
} else throw new Error(`Unexpected agent command: ${kind}`);
return result as never;
});
}
+33
View File
@@ -0,0 +1,33 @@
import { createHash, randomUUID } from 'node:crypto';
/** Seed verified host inventory without copying a complete game distribution. */
export async function prepareNativeHost(servers: string[] = []) {
const { config } = await import('../../src/lib/server/config.js');
const hosts = await import('../../src/lib/server/hosts/index.js');
const libraries = await import('../../src/lib/server/game-files/index.js');
const { db } = await import('../../src/lib/server/db/index.js');
const host = hosts.createHost({ name: 'Fixture host', portMin: 28960, portMax: 29060 });
const inventory = { protocolVersion: 1, cpuCount: 64, cpuUsagePercent: 0,
memoryTotal: 128 * 1024 ** 3, memoryAvailable: 120 * 1024 ** 3,
dockerRunning: true, directoriesWritable: true, storageAvailable: 100_000_000_000,
runtimeImages: [config.defaultRuntimeImage] };
hosts.enrollHost(hosts.createEnrollment(host.id).token, inventory);
const versions = new Map<string, string>();
for (const game of ['t5', 't6'] as const) {
const version = libraries.createLibraryImport({ hostId: host.id, game });
const manifest = JSON.stringify([{ path: 'zone/base.ff', size: 4, sha256: createHash('sha256').update('game').digest('hex') }]);
const hash = createHash('sha256').update(manifest).digest('hex');
db().prepare("UPDATE game_file_versions SET state='ready',manifest=?,manifest_sha256=?,size_bytes=4,published_at=? WHERE id=?")
.run(manifest, hash, new Date().toISOString(), version.id);
libraries.updateHostLibraryCaches(host.id, [{ libraryVersionId: version.id, status: 'ready', manifestSha256: hash }]);
versions.set(game, version.id);
}
const heartbeat = () => hosts.heartbeat(host.id, { ...inventory, libraries: [...versions.values()].map(id => ({ id, state: 'ready' as const })) });
heartbeat();
for (const [index, server] of servers.entries()) {
db().prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at)
VALUES(?,?,?,?,?,?,?)`).run(randomUUID(), server, host.id, versions.get(server.startsWith('t6_') ? 't6' : 't5'),
config.defaultRuntimeImage, 28960 + index, new Date().toISOString());
}
return { host, versions, heartbeat };
}
+9 -100
View File
@@ -1,17 +1,9 @@
import { prepareNativeHost } from '../helpers/native-host.js';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { access } from 'node:fs/promises'; import { access } from 'node:fs/promises';
import { join } from 'node:path'; import { join } from 'node:path';
import { newFixture, type Fixture } from '../helpers/fixture.js'; import { newFixture, type Fixture } from '../helpers/fixture.js';
vi.mock('../../src/lib/server/integrations/komodo.js', () => ({
komodoCreateDeployment: vi.fn(async () => ({ ok: true })),
komodoDeploy: vi.fn(async () => ({ ok: true })),
komodoDestroyDeployment: vi.fn(async () => ({ ok: true })),
komodoDeleteDeployment: vi.fn(async () => ({ ok: true })),
komodoInspectHealthy: vi.fn(),
komodoEnsureTag: vi.fn(async (name: string) => ({ ok: true, id: name })),
komodoUpdateTagsOnResource: vi.fn(async () => ({ ok: true }))
}));
vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ vi.mock('../../src/lib/server/integrations/rcon.js', () => ({
rconGetPlayerCount: vi.fn(), rconStatusResult: vi.fn() rconGetPlayerCount: vi.fn(), rconStatusResult: vi.fn()
})); }));
@@ -24,19 +16,18 @@ const server = 't5_zombie_theater_1';
beforeEach(async () => { beforeEach(async () => {
vi.resetModules(); vi.clearAllMocks(); vi.resetModules(); vi.clearAllMocks();
fixture = await newFixture([otherServer, server]); fixture = await newFixture([otherServer, server]);
for (const [key, value] of Object.entries({ KOMODO_ADDRESS: 'http://komodo.test', KOMODO_API_KEY: 'test', vi.stubEnv('RCON_PASSWORD', 'test');
KOMODO_API_SECRET: 'test', KOMODO_TARGET_SERVER: 'test-host', RCON_PASSWORD: 'test',
SERVER_PORT_MIN: '28960', SERVER_PORT_MAX: '28980' })) vi.stubEnv(key, value);
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
for (const [index, name] of [otherServer, server].entries()) { for (const [index, name] of [otherServer, server].entries()) {
const map = name === server ? 'theater' : 'cosmodrome'; const map = name === server ? 'theater' : 'cosmodrome';
db().prepare(`INSERT INTO servers (name, komodo_type, komodo_name, game, map, game_mode, map_slug, game_port, host_dir) db().prepare(`INSERT INTO servers (name, game, map, game_mode, map_slug, game_port, host_dir)
VALUES (?, 'deployment', ?, 't5', ?, 'zombie', ?, ?, ?)`).run(name, name, 'zombie_' + map, map, 28960 + index, join(fixture.t5, name)); VALUES (?, 't5', ?, 'zombie', ?, ?, ?)`).run(name, 'zombie_' + map, map, 28960 + index, join(fixture.t5, name));
} }
await prepareNativeHost([otherServer, server]);
const { mockAgentFiles } = await import('../helpers/agent-files.js');
await mockAgentFiles();
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js'); const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(rconGetPlayerCount).mockReset().mockResolvedValue(0); vi.mocked(rconGetPlayerCount).mockReset().mockResolvedValue(0);
vi.mocked(komodoInspectHealthy).mockReset().mockResolvedValue({ ok: false, state: 'exited', reason: 'stopped' });
}); });
afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); }); afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); });
@@ -55,9 +46,7 @@ async function attachBeta(state = 'running', outcome?: string, deleted = false)
async function expectServerPreserved() { async function expectServerPreserved() {
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const { komodoDestroyDeployment, komodoDeleteDeployment } = await import('../../src/lib/server/integrations/komodo.js'); expect(db().prepare("SELECT id FROM ec_host_operations WHERE kind LIKE 'instance.%'").all()).toEqual([]);
expect(komodoDestroyDeployment).not.toHaveBeenCalled();
expect(komodoDeleteDeployment).not.toHaveBeenCalled();
expect(db().prepare('SELECT name FROM servers WHERE name = ?').get(server)).toEqual({ name: server }); expect(db().prepare('SELECT name FROM servers WHERE name = ?').get(server)).toEqual({ name: server });
await expect(access(join(fixture.t5, server))).resolves.toBeUndefined(); await expect(access(join(fixture.t5, server))).resolves.toBeUndefined();
} }
@@ -117,86 +106,6 @@ describe('active beta destruction guard', () => {
expect(destroyServer(server)).toMatchObject({ ok: true, deploymentId: expect.any(String) }); expect(destroyServer(server)).toMatchObject({ ok: true, deploymentId: expect.any(String) });
}); });
it('holds a queued destruction until its beta closes, then completes the same request', async () => {
const { destroyServer, processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { getDeployment } = await import('../../src/lib/server/fleet/deployments.js');
const { db } = await import('../../src/lib/server/db/index.js');
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
const { komodoDestroyDeployment, komodoDeleteDeployment } = await import('../../src/lib/server/integrations/komodo.js');
const queued = destroyServer(server);
expect(queued.ok).toBe(true);
const id = await attachBeta();
await processDestructions();
expect(getDeployment(queued.deploymentId!)?.deployment).toMatchObject({ state: 'waiting', stage: expect.stringMatching(/beta/i) });
expect(rconGetPlayerCount).not.toHaveBeenCalled();
await expectServerPreserved();
db().prepare("UPDATE betas SET state = 'rolled-back', closed_at = ? WHERE id = ?").run(new Date().toISOString(), id);
await processDestructions();
expect(komodoDestroyDeployment).toHaveBeenCalledOnce();
expect(komodoDeleteDeployment).toHaveBeenCalledOnce();
expect(getDeployment(queued.deploymentId!)?.deployment).toMatchObject({ state: 'done', server_exists: 0 });
});
it('rechecks beta ownership after an in-flight empty-server probe', async () => {
const { destroyServer, processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { getDeployment } = await import('../../src/lib/server/fleet/deployments.js');
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
let resolveCount!: (players: number) => void;
vi.mocked(rconGetPlayerCount).mockImplementationOnce(() => new Promise(resolve => { resolveCount = resolve; }));
const queued = destroyServer(server);
const work = processDestructions();
await vi.waitFor(() => expect(resolveCount).toBeTypeOf('function'));
await attachBeta();
resolveCount(0);
await work;
expect(getDeployment(queued.deploymentId!)?.deployment).toMatchObject({ state: 'waiting', stage: expect.stringMatching(/beta/i) });
await expectServerPreserved();
});
it('also holds rebuild destruction when a beta appears during the occupancy probe', async () => {
const { requestLifecycle, processLifecycle } = await import('../../src/lib/server/fleet/lifecycle.js');
const { saveServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
const { getDeployment } = await import('../../src/lib/server/fleet/deployments.js');
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
saveServerKey(server, 'saved-key');
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: true, state: 'running' });
let resolveCount!: (players: number) => void;
vi.mocked(rconGetPlayerCount).mockImplementationOnce(() => new Promise(resolve => { resolveCount = resolve; }));
const queued = requestLifecycle(server, 'rebuild');
if (!queued.ok) throw new Error(queued.error);
const work = processLifecycle();
await vi.waitFor(() => expect(resolveCount).toBeTypeOf('function'));
await attachBeta();
resolveCount(0);
await work;
expect(getDeployment(queued.deploymentId)?.deployment).toMatchObject({ state: 'waiting', stage: expect.stringMatching(/beta/i) });
await expectServerPreserved();
});
it('preserves the cleanup checkpoint and files while an active beta blocks a legacy cleanup resume', async () => {
const { destroyServer, processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { deploymentStage, finishDeployment, resumeCleanup, getDeployment, REMOVING_SERVER } = await import('../../src/lib/server/fleet/deployments.js');
const { db } = await import('../../src/lib/server/db/index.js');
const queued = destroyServer(server);
deploymentStage(queued.deploymentId!, REMOVING_SERVER);
finishDeployment(queued.deploymentId!, { ok: false, message: 'Interrupted local cleanup' });
const id = await attachBeta();
expect(resumeCleanup(queued.deploymentId!)).toBe(true);
await processDestructions();
expect(getDeployment(queued.deploymentId!)?.deployment).toMatchObject({ state: 'failed', stage: REMOVING_SERVER,
error: expect.stringMatching(/beta/i) });
expect(getDeployment(queued.deploymentId!)?.canResumeCleanup).toBe(true);
await expectServerPreserved();
db().prepare("UPDATE betas SET state = 'promoted' WHERE id = ?").run(id);
expect(resumeCleanup(queued.deploymentId!)).toBe(true);
await processDestructions();
expect(getDeployment(queued.deploymentId!)?.deployment).toMatchObject({ state: 'done', server_exists: 0 });
const { komodoDestroyDeployment, komodoDeleteDeployment } = await import('../../src/lib/server/integrations/komodo.js');
expect(komodoDestroyDeployment).not.toHaveBeenCalled();
expect(komodoDeleteDeployment).not.toHaveBeenCalled();
});
it.each(['destroy', 'rebuild'] as const)('requires cancelling a pending %s before opening a beta', async action => { it.each(['destroy', 'rebuild'] as const)('requires cancelling a pending %s before opening a beta', async action => {
const { destroyServer } = await import('../../src/lib/server/fleet/create.js'); const { destroyServer } = await import('../../src/lib/server/fleet/create.js');
const { requestLifecycle } = await import('../../src/lib/server/fleet/lifecycle.js'); const { requestLifecycle } = await import('../../src/lib/server/fleet/lifecycle.js');
@@ -222,7 +131,7 @@ describe('active beta destruction guard', () => {
}); });
it.each((['destroy', 'rebuild'] as const).flatMap(action => ['failed', 'interrupted'].flatMap(state => it.each((['destroy', 'rebuild'] as const).flatMap(action => ['failed', 'interrupted'].flatMap(state =>
['Stopping and removing container', 'Deleting Komodo deployment', 'Removing server files and registry'].map(stage => ({ action, state, stage })) ['Stopping and removing container', 'Removing server files and registry'].map(stage => ({ action, state, stage }))
)))('rejects beta admission after $action is $state at $stage', async ({ action, state, stage }) => { )))('rejects beta admission after $action is $state at $stage', async ({ action, state, stage }) => {
const { beginDeployment } = await import('../../src/lib/server/fleet/deployments.js'); const { beginDeployment } = await import('../../src/lib/server/fleet/deployments.js');
const { createScript } = await import('../../src/lib/server/fleet/scripts.js'); const { createScript } = await import('../../src/lib/server/fleet/scripts.js');
+21 -9
View File
@@ -6,9 +6,26 @@ import { newFixture, type Fixture } from '../helpers/fixture.js';
import { gameEvent, postTelemetry } from '../helpers/telemetry.js'; import { gameEvent, postTelemetry } from '../helpers/telemetry.js';
import { defaultPlan, requirement, type BetaPlan } from '../../src/lib/beta-plan.js'; import { defaultPlan, requirement, type BetaPlan } from '../../src/lib/beta-plan.js';
vi.mock('../../src/lib/server/integrations/komodo.js', () => ({ komodoRestart: vi.fn(), komodoInspectHealthy: vi.fn() }));
vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconGetPlayerCount: vi.fn(), rconStatusResult: vi.fn() })); vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconGetPlayerCount: vi.fn(), rconStatusResult: vi.fn() }));
vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn() })); vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn() }));
// Planner tests observe the provisioning contract. Host command delivery and
// disruption safety are exercised by the native provisioning suites.
vi.mock('../../src/lib/server/provisioning/worker.js', () => ({
processProvisioning: async () => {
const { db } = await import('../../src/lib/server/db/index.js');
const { applyStagedFilesForOperation } = await import('../../src/lib/server/fleet/staged-files.js');
const { probePlayerEvidence } = await import('../../src/lib/server/fleet/status-evidence.js');
const jobs = db().prepare("SELECT id,server,state FROM server_deployments WHERE action='restart' AND state IN ('waiting','awaiting-health')").all() as { id: string; server: string; state: string }[];
for (const job of jobs) {
if (job.state === 'awaiting-health') {
db().prepare("UPDATE server_deployments SET state='done',finished_at=? WHERE id=?").run(new Date().toISOString(), job.id);
} else if ((await probePlayerEvidence(job.server)).players === 0) {
applyStagedFilesForOperation(job.id, job.server);
db().prepare("UPDATE server_deployments SET state='awaiting-health',submitted_at=? WHERE id=?").run(new Date().toISOString(), job.id);
}
}
}
}));
let fixture: Fixture; let fixture: Fixture;
const names = ['t5_zombie_theater_1', 't5_zombie_theater_2']; const names = ['t5_zombie_theater_1', 't5_zombie_theater_2'];
const now = () => new Date().toISOString(); const now = () => new Date().toISOString();
@@ -21,14 +38,11 @@ beforeEach(async () => {
await writeFile(join(fixture.t5, name, 'logs/console.log'), 'old script error\n'); await writeFile(join(fixture.t5, name, 'logs/console.log'), 'old script error\n');
await writeFile(join(fixture.t5, name, 'logs/games_sp.log'), 'InitGame:\n 0:00 J;old;1;old\nExitLevel: executed\n'); await writeFile(join(fixture.t5, name, 'logs/games_sp.log'), 'InitGame:\n 0:00 J;old;1;old\nExitLevel: executed\n');
} }
vi.stubEnv('KOMODO_ADDRESS', 'http://komodo.test'); vi.stubEnv('KOMODO_API_KEY', 'test');
vi.stubEnv('KOMODO_API_SECRET', 'test'); vi.stubEnv('RCON_PASSWORD', 'test'); vi.stubEnv('DEPLOYMENT_RESTART_CONCURRENCY', '2'); vi.stubEnv('RCON_PASSWORD', 'test'); vi.stubEnv('DEPLOYMENT_RESTART_CONCURRENCY', '2');
const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js'); const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js');
await seedServerRegistry(); await seedServerRegistry();
const { komodoRestart, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
const { rconGetPlayerCount, rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js'); const { rconGetPlayerCount, rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js');
vi.mocked(komodoRestart).mockResolvedValue({ ok: true });
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: true, state: 'running' });
vi.mocked(rconGetPlayerCount).mockResolvedValue(0); vi.mocked(rconGetPlayerCount).mockResolvedValue(0);
vi.mocked(rconStatusResult).mockResolvedValue({ ok: true, endpoint: { host: 'test', port: 1 }, rawReply: '', status: { players: 0, map: 'zombie_theater', gametype: 'zom', roster: [] } }); vi.mocked(rconStatusResult).mockResolvedValue({ ok: true, endpoint: { host: 'test', port: 1 }, rawReply: '', status: { players: 0, map: 'zombie_theater', gametype: 'zom', roster: [] } });
}); });
@@ -57,14 +71,12 @@ describe('durable beta test plans', () => {
it('waits for occupied telemetry to clear before restarting a beta despite an empty RCON response', async () => { it('waits for occupied telemetry to clear before restarting a beta despite an empty RCON response', async () => {
const id = await openPlan(defaultPlan()); const id = await openPlan(defaultPlan());
const { listBetaServers } = await import('../../src/lib/server/fleet/beta.js'); const { listBetaServers } = await import('../../src/lib/server/fleet/beta.js');
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
await postTelemetry(gameEvent('heartbeat', 4, 2)); await postTelemetry(gameEvent('heartbeat', 4, 2));
await tick(); await tick();
expect(komodoRestart).not.toHaveBeenCalled();
expect(listBetaServers(id)[0].stage).toBe('awaiting-empty'); expect(listBetaServers(id)[0].stage).toBe('awaiting-empty');
await postTelemetry(gameEvent('heartbeat', 4, 0)); await postTelemetry(gameEvent('heartbeat', 4, 0));
await tick(); await tick();
expect(komodoRestart).toHaveBeenCalledTimes(1); expect(listBetaServers(id)[0].stage).toBe('awaiting-health');
}); });
it('holds file deployment until each wave passes, and rollback leaves undeployed files alone', async () => { it('holds file deployment until each wave passes, and rollback leaves undeployed files alone', async () => {
+1 -1
View File
@@ -9,7 +9,7 @@ let fixture: Fixture;
const servers = ['t5_zombie_theater_1', 't5_zombie_cosmodrome_1']; const servers = ['t5_zombie_theater_1', 't5_zombie_cosmodrome_1'];
beforeEach(async () => { beforeEach(async () => {
vi.resetModules(); fixture = await newFixture(servers); vi.resetModules(); fixture = await newFixture(servers);
vi.stubEnv('KOMODO_ADDRESS', ''); vi.stubEnv('RCON_PASSWORD', ''); vi.stubEnv('RCON_PASSWORD', '');
const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js'); const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js');
await seedServerRegistry(); await seedServerRegistry();
}); });
+1 -1
View File
@@ -188,7 +188,7 @@ describe('staged beta files', () => {
const { requestRestart } = await import('../../src/lib/server/provisioning/index.js'); const { requestRestart } = await import('../../src/lib/server/provisioning/index.js');
const first = requestRestart(servers[1], { kind: 'campaign', id: older.deploymentId }); const first = requestRestart(servers[1], { kind: 'campaign', id: older.deploymentId });
const second = requestRestart(servers[1], { kind: 'campaign', id: latest.deploymentId }); const second = requestRestart(servers[1], { kind: 'campaign', id: latest.deploymentId });
expect(second.id).toBe(first.id); expect(second.id).not.toBe(first.id);
await activateFiles('campaign', latest.deploymentId); await activateFiles('campaign', latest.deploymentId);
const { destOfKind, configFileOf } = await import('../../src/lib/server/config.js'); const { destOfKind, configFileOf } = await import('../../src/lib/server/config.js');
const relPath = kind === 'script' ? script.target_rel_path : configFileOf(servers[1]); const relPath = kind === 'script' ? script.target_rel_path : configFileOf(servers[1]);
@@ -18,9 +18,7 @@ beforeEach(async () => {
vi.stubEnv('GAME_FILES_ROOT', join(fixture.root, 'libraries')); vi.stubEnv('GAME_FILES_ROOT', join(fixture.root, 'libraries'));
vi.stubEnv('PLUTAINER_IMAGE', image); vi.stubEnv('PLUTAINER_IMAGE', image);
vi.stubEnv('RCON_PASSWORD', 'test-password'); vi.stubEnv('RCON_PASSWORD', 'test-password');
vi.stubEnv('KOMODO_ADDRESS', '');
vi.stubEnv('KOMODO_API_KEY', '');
vi.stubEnv('KOMODO_API_SECRET', '');
const { createHost, createEnrollment, enrollHost, heartbeat } = await import('../../src/lib/server/hosts/index.js'); const { createHost, createEnrollment, enrollHost, heartbeat } = await import('../../src/lib/server/hosts/index.js');
hostId = createHost({ name: 'Large native fleet', portMin: 28960, portMax: 29059 }).id; hostId = createHost({ name: 'Large native fleet', portMin: 28960, portMax: 29059 }).id;
const inventory = { const inventory = {
@@ -24,9 +24,7 @@ beforeEach(async () => {
vi.stubEnv('GAME_FILES_ROOT', join(fixture.root, 'libraries')); vi.stubEnv('GAME_FILES_ROOT', join(fixture.root, 'libraries'));
vi.stubEnv('PLUTAINER_IMAGE', image); vi.stubEnv('PLUTAINER_IMAGE', image);
vi.stubEnv('RCON_PASSWORD', 'test-password'); vi.stubEnv('RCON_PASSWORD', 'test-password');
vi.stubEnv('KOMODO_ADDRESS', '');
vi.stubEnv('KOMODO_API_KEY', '');
vi.stubEnv('KOMODO_API_SECRET', '');
vi.stubGlobal('fetch', vi.fn(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354, files: [] }))); vi.stubGlobal('fetch', vi.fn(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354, files: [] })));
const { createHost, createEnrollment, enrollHost, heartbeat, reserveHostPort } = await import('../../src/lib/server/hosts/index.js'); const { createHost, createEnrollment, enrollHost, heartbeat, reserveHostPort } = await import('../../src/lib/server/hosts/index.js');
hostId = createHost({ name: 'Match collection host', portMin: 28960, portMax: 28999 }).id; hostId = createHost({ name: 'Match collection host', portMin: 28960, portMax: 28999 }).id;
@@ -55,8 +53,8 @@ beforeEach(async () => {
const instanceId = randomUUID(); const instanceId = randomUUID();
const deploymentId = randomUUID(); const deploymentId = randomUUID();
const port = 28960 + index; const port = 28960 + index;
db().prepare(`INSERT INTO servers(name,komodo_name,game,map,game_mode,map_slug,game_port,container_state,last_players,last_seen_at) db().prepare(`INSERT INTO servers(name,game,map,game_mode,map_slug,game_port,container_state,last_players,last_seen_at)
VALUES(?,'','t6',?,'zombie',?,?,'running',0,?)`).run(name, map.canonical, map.slug, port, now); VALUES(?,'t6',?,'zombie',?,?,'running',0,?)`).run(name, map.canonical, map.slug, port, now);
db().prepare('INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at) VALUES(?,?,?,?,?,?,?)') db().prepare('INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at) VALUES(?,?,?,?,?,?,?)')
.run(instanceId, name, hostId, libraries[1].id, image, port, now); .run(instanceId, name, hostId, libraries[1].id, image, port, now);
db().prepare("INSERT INTO server_deployments(id,server,action,state,stage,triggered_by,started_at,finished_at) VALUES(?,?,'create','done','Verified','fixture',?,?)") db().prepare("INSERT INTO server_deployments(id,server,action,state,stage,triggered_by,started_at,finished_at) VALUES(?,?,'create','done','Verified','fixture',?,?)")
+1 -1
View File
@@ -10,7 +10,7 @@ let fixture: Fixture;
beforeEach(async () => { beforeEach(async () => {
vi.resetModules(); vi.resetModules();
fixture = await newFixture(['t5_zombie_theater_1']); fixture = await newFixture(['t5_zombie_theater_1']);
delete process.env.KOMODO_ADDRESS;
delete process.env.RCON_PASSWORD; delete process.env.RCON_PASSWORD;
}); });
-803
View File
@@ -1,803 +0,0 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { existsSync } from 'node:fs';
import { readFile, rm } from 'node:fs/promises';
import { join } from 'node:path';
import { newFixture, type Fixture } from '../helpers/fixture.js';
import { T6_ZOMBIE_MAPS } from '../../src/lib/map-aliases.js';
vi.mock('../../src/lib/server/integrations/komodo.js', () => ({
komodoCreateDeployment: vi.fn(async () => ({ ok: true })),
komodoDeploy: vi.fn(async () => ({ ok: true })),
komodoDestroyDeployment: vi.fn(async () => ({ ok: true })),
komodoDeleteDeployment: vi.fn(async () => ({ ok: true })),
komodoInspectHealthy: vi.fn(),
komodoEnsureTag: vi.fn(async (name: string) => ({ ok: true, id: name })),
komodoUpdateTagsOnResource: vi.fn(async () => ({ ok: true }))
}));
vi.mock('../../src/lib/server/integrations/rcon.js', () => ({
rconGetPlayerCount: vi.fn(async () => 0),
rconPlutoniumVersion: vi.fn(async () => ({ revision: 5354, error: null })),
rconStatusResult: vi.fn(async () => ({ ok: true, endpoint: { host: 'localhost', port: 27016 }, rawReply: '',
status: { players: 0, map: 'zombie_cosmodrome', gametype: null, roster: [] } }))
}));
vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}), kickProvisioning: vi.fn(async () => {}) }));
let fixture: Fixture;
const serverName = 't5_zombie_cosmodrome_1';
const input = { gameMode: 'zombie' as const, mapSlug: 'cosmodrome', plutoKey: 'test-key' };
beforeEach(async () => {
vi.resetModules();
vi.clearAllMocks();
fixture = await newFixture([]);
vi.stubGlobal('fetch', vi.fn(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354, files: [] })));
vi.stubEnv('KOMODO_ADDRESS', 'http://komodo.test');
vi.stubEnv('KOMODO_API_KEY', 'test');
vi.stubEnv('KOMODO_API_SECRET', 'test');
vi.stubEnv('KOMODO_TARGET_SERVER', 'test-host');
vi.stubEnv('RCON_PASSWORD', 'test-rcon');
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
const { rconStatusResult, rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
vi.mocked(komodoInspectHealthy).mockReset().mockResolvedValue({ ok: true, state: 'running' });
vi.mocked(rconStatusResult).mockReset().mockResolvedValue({ ok: true,
endpoint: { host: 'localhost', port: 27016 }, rawReply: '',
status: { players: 0, map: 'zombie_cosmodrome', gametype: null, roster: [] } });
vi.mocked(rconGetPlayerCount).mockReset().mockResolvedValue(0);
});
afterEach(async () => {
await fixture.cleanup();
vi.unstubAllEnvs();
vi.unstubAllGlobals();
});
describe('queued removal cancellation', () => {
it('lets cancellation win while the worker awaits a player count, and never removes the server', async () => {
const { createServer, destroyServer, processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { getDeployment } = await import('../../src/lib/server/fleet/deployments.js');
const { POST } = await import('../../src/routes/api/deployments/[id]/cancel/+server.js');
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
const { komodoDestroyDeployment } = await import('../../src/lib/server/integrations/komodo.js');
await createServer(input);
let resolveCount!: (players: number) => void;
vi.mocked(rconGetPlayerCount).mockImplementationOnce(() => new Promise(resolve => { resolveCount = resolve; }));
const queued = destroyServer(serverName);
const work = processDestructions();
await vi.waitFor(() => expect(resolveCount).toBeTypeOf('function'));
expect((await POST({ params: { id: queued.deploymentId! } })).status).toBe(200);
resolveCount(0);
await work;
expect(komodoDestroyDeployment).not.toHaveBeenCalled();
expect(getDeployment(queued.deploymentId!)?.deployment).toMatchObject({ state: 'cancelled', server_exists: 1, stage: 'Cancelled by operator' });
expect(getDeployment(queued.deploymentId!)?.canCancelRemoval).toBe(false);
expect((await POST({ params: { id: queued.deploymentId! } })).status).toBe(200);
});
it('does not overwrite cancellation when the in-flight player check rejects', async () => {
const { createServer, destroyServer, processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { cancelRemoval, getDeployment } = await import('../../src/lib/server/fleet/deployments.js');
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
await createServer(input);
let rejectCount!: (error: Error) => void;
vi.mocked(rconGetPlayerCount).mockImplementationOnce(() => new Promise((_resolve, reject) => { rejectCount = reject; }));
const queued = destroyServer(serverName);
const work = processDestructions();
await vi.waitFor(() => expect(rejectCount).toBeTypeOf('function'));
expect(cancelRemoval(queued.deploymentId!)).toBe(true);
rejectCount(new Error('RCON unavailable'));
await work;
expect(getDeployment(queued.deploymentId!)?.deployment).toMatchObject({ state: 'cancelled', error: null });
});
it('refuses cancellation after an atomic removal claim and for cleanup checkpoints', async () => {
const { createServer, destroyServer } = await import('../../src/lib/server/fleet/create.js');
const { cancelRemoval, claimRemoval, canCancelRemoval, deploymentStage, REMOVING_SERVER } = await import('../../src/lib/server/fleet/deployments.js');
const { db } = await import('../../src/lib/server/db/index.js');
await createServer(input);
const queued = destroyServer(serverName);
expect(claimRemoval(queued.deploymentId!)).toBe(true);
expect(cancelRemoval(queued.deploymentId!)).toBe(false);
expect(claimRemoval(queued.deploymentId!)).toBe(false);
deploymentStage(queued.deploymentId!, REMOVING_SERVER);
db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(queued.deploymentId);
expect(canCancelRemoval(queued.deploymentId!)).toBe(false);
expect(cancelRemoval(queued.deploymentId!)).toBe(false);
});
});
describe('destruction cleanup recovery', () => {
it('removes T6 servers while preserving closed beta progress and test evidence', async () => {
const { createServer, destroyServer, processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { createScript } = await import('../../src/lib/server/fleet/scripts.js');
const { db } = await import('../../src/lib/server/db/index.js');
const { getDeployment } = await import('../../src/lib/server/fleet/deployments.js');
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
const { rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js');
vi.mocked(rconStatusResult).mockResolvedValueOnce({ ok: true,
endpoint: { host: 'localhost', port: 28960 }, rawReply: '',
status: { players: 0, map: 'zm_transit', gametype: null, roster: [] } });
const name = 't6_zombie_town_1';
expect(await createServer({ ...input, game: 't6', mapSlug: 'town' })).toMatchObject({ ok: true, name });
expect(await createServer(input)).toMatchObject({ ok: true });
const { script, version } = createScript({ name: 'test script', kind: 'script', scopeType: 'game', scopeValue: 't6', targetRelPath: 'test.gsc', content: Buffer.from('main(){}') });
const beta = db().prepare("INSERT INTO betas (script_id, script_version_id, trigger_kind, state) VALUES (?, ?, 'manual', 'promoted')").run(script.id, version.id).lastInsertRowid;
for (const server of [name, serverName]) {
db().prepare("INSERT INTO beta_servers (beta_id, server, had_original, stage, healthy_at) VALUES (?, ?, 0, 'ready', '2026-09-18')").run(beta, server);
db().prepare("INSERT INTO beta_test_runs (id, beta_id, server, created_at) VALUES (?, ?, ?, '2026-09-18')").run(server, beta, server);
db().prepare("INSERT INTO beta_test_evidence (run_id, event_id, received_at, kind, payload) VALUES (?, 'event', '2026-09-18', 'health', '{}')").run(server);
db().prepare('INSERT INTO beta_test_coverage (beta_id, server) VALUES (?, ?)').run(beta, server);
db().prepare("INSERT INTO beta_file_changes (beta_id, server, rel_path) VALUES (?, ?, 'test.gsc')").run(beta, server);
}
const queued = destroyServer(name);
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, reason: 'gone' });
await processDestructions();
expect(getDeployment(queued.deploymentId!)?.deployment).toMatchObject({ state: 'done', server_exists: 0 });
for (const table of ['beta_servers', 'beta_test_runs', 'beta_test_coverage', 'beta_file_changes']) {
expect(db().prepare(`SELECT server FROM ${table} ORDER BY server`).all()).toEqual([{ server: serverName }, { server: name }]);
}
expect(db().prepare('SELECT run_id FROM beta_test_evidence ORDER BY run_id').all()).toEqual([{ run_id: serverName }, { run_id: name }]);
expect(db().prepare('SELECT stage, healthy_at, server_deleted_at FROM beta_servers WHERE server = ?').get(name))
.toEqual({ stage: 'ready', healthy_at: '2026-09-18', server_deleted_at: expect.any(String) });
expect(db().pragma('foreign_key_check')).toEqual([]);
});
it('resumes failed local cleanup after restart without RCON or another Komodo deletion', async () => {
const { createServer, destroyServer, processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { getDeployment, interruptDeployments } = await import('../../src/lib/server/fleet/deployments.js');
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
const { komodoInspectHealthy, komodoDestroyDeployment, komodoDeleteDeployment } = await import('../../src/lib/server/integrations/komodo.js');
const { POST } = await import('../../src/routes/api/deployments/[id]/resume/+server.js');
await createServer(input);
db().exec("CREATE TRIGGER fail_cleanup BEFORE DELETE ON servers BEGIN SELECT RAISE(ABORT, 'FOREIGN KEY constraint failed'); END");
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, reason: 'gone' });
const { deploymentId } = destroyServer(serverName);
await processDestructions();
expect(getDeployment(deploymentId!)?.deployment).toMatchObject({ state: 'failed', stage: 'Removing server files and registry', server_exists: 1 });
expect(getDeployment(deploymentId!)?.canResumeCleanup).toBe(true);
db().exec('DROP TRIGGER fail_cleanup');
closeDb();
interruptDeployments();
vi.mocked(rconGetPlayerCount).mockClear();
vi.mocked(komodoDestroyDeployment).mockClear();
vi.mocked(komodoDeleteDeployment).mockClear();
expect((await POST({ params: { id: deploymentId! } })).status).toBe(202);
expect((await POST({ params: { id: deploymentId! } })).status).toBe(409);
await processDestructions();
expect(rconGetPlayerCount).not.toHaveBeenCalled();
expect(komodoDestroyDeployment).not.toHaveBeenCalled();
expect(komodoDeleteDeployment).not.toHaveBeenCalled();
expect(getDeployment(deploymentId!)?.deployment).toMatchObject({ state: 'done', server_exists: 0 });
});
it('reuses an interrupted cleanup when destruction is requested again', async () => {
const { createServer, destroyServer } = await import('../../src/lib/server/fleet/create.js');
const { deploymentStage, interruptDeployments, getDeployment, REMOVING_SERVER } = await import('../../src/lib/server/fleet/deployments.js');
const { db } = await import('../../src/lib/server/db/index.js');
await createServer(input);
const first = destroyServer(serverName);
deploymentStage(first.deploymentId!, REMOVING_SERVER);
db().prepare("UPDATE server_deployments SET state = 'running' WHERE id = ?").run(first.deploymentId);
interruptDeployments();
expect(destroyServer(serverName).deploymentId).toBe(first.deploymentId);
expect(getDeployment(first.deploymentId!)?.deployment).toMatchObject({ state: 'waiting', stage: REMOVING_SERVER });
});
it('resumes cleanup after crashing between registry deletion and the final result', async () => {
const { createServer, destroyServer, processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { claimRemoval, deploymentStage, interruptDeployments, getDeployment, resumeCleanup, REMOVING_SERVER } = await import('../../src/lib/server/fleet/deployments.js');
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
const { komodoDestroyDeployment, komodoDeleteDeployment } = await import('../../src/lib/server/integrations/komodo.js');
await createServer(input);
const { deploymentId } = destroyServer(serverName);
claimRemoval(deploymentId!);
deploymentStage(deploymentId!, REMOVING_SERVER);
db().prepare('DELETE FROM servers WHERE name = ?').run(serverName);
closeDb();
interruptDeployments();
expect(getDeployment(deploymentId!)?.deployment).toMatchObject({ state: 'interrupted', server_deleted_at: expect.any(String) });
expect(resumeCleanup(deploymentId!)).toBe(true);
await processDestructions();
expect(getDeployment(deploymentId!)?.deployment).toMatchObject({ state: 'done', server_deleted_at: expect.any(String), server_exists: 0 });
expect(komodoDestroyDeployment).not.toHaveBeenCalled();
expect(komodoDeleteDeployment).not.toHaveBeenCalled();
});
it.each(['before', 'after'] as const)('protects a replacement registered %s an archived cleanup resume', async timing => {
const { createServer, destroyServer, processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { claimRemoval, deploymentStage, interruptDeployments, resumeCleanup, REMOVING_SERVER } = await import('../../src/lib/server/fleet/deployments.js');
const { db } = await import('../../src/lib/server/db/index.js');
await createServer(input);
const { deploymentId } = destroyServer(serverName);
claimRemoval(deploymentId!);
deploymentStage(deploymentId!, REMOVING_SERVER);
db().prepare('DELETE FROM servers WHERE name = ?').run(serverName);
interruptDeployments();
if (timing === 'after') expect(resumeCleanup(deploymentId!)).toBe(true);
db().prepare('INSERT INTO servers (name, komodo_name) VALUES (?, ?)').run(serverName, 'replacement');
if (timing === 'before') expect(resumeCleanup(deploymentId!)).toBe(false);
await processDestructions();
expect(db().prepare('SELECT komodo_name FROM servers WHERE name = ?').get(serverName)).toEqual({ komodo_name: 'replacement' });
});
it('refuses cleanup before confirmed deletion or after a newer creation', async () => {
const { createServer, destroyServer } = await import('../../src/lib/server/fleet/create.js');
const { beginDeployment, deploymentStage, finishDeployment, resumeCleanup, REMOVING_SERVER } = await import('../../src/lib/server/fleet/deployments.js');
await createServer(input);
const { deploymentId } = destroyServer(serverName);
finishDeployment(deploymentId!, { ok: false, message: 'Komodo unavailable' });
expect(resumeCleanup(deploymentId!)).toBe(false);
deploymentStage(deploymentId!, REMOVING_SERVER);
beginDeployment(serverName, 'create');
expect(resumeCleanup(deploymentId!)).toBe(false);
});
});
describe('House Party Protocol', () => {
it('includes registered and queued servers once and reuses destruction requests', async () => {
const { createServer, queueCreateServer } = await import('../../src/lib/server/fleet/create.js');
const { housePartyTargets, queueHouseParty } = await import('../../src/lib/server/fleet/house-party.js');
expect(await createServer(input)).toMatchObject({ ok: true });
expect(queueCreateServer(input)).toMatchObject({ ok: true });
const names = housePartyTargets();
expect(names).toEqual([serverName, 't5_zombie_cosmodrome_2']);
const first = queueHouseParty(names);
expect(first).toMatchObject({ ok: true, deployments: [{ server: serverName }, { server: names[1] }] });
expect(queueHouseParty(names)).toEqual(first);
});
it('rejects a changed fleet without queuing any destruction', async () => {
const { queueCreateServer } = await import('../../src/lib/server/fleet/create.js');
const { housePartyTargets, queueHouseParty } = await import('../../src/lib/server/fleet/house-party.js');
const { db } = await import('../../src/lib/server/db/index.js');
const names = housePartyTargets();
expect(queueCreateServer(input).ok).toBe(true);
expect(queueHouseParty(names).ok).toBe(false);
expect(db().prepare("SELECT * FROM server_deployments WHERE action = 'destroy'").all()).toEqual([]);
});
it('rolls back the entire bulk request when an insert fails', async () => {
const { queueCreateServer } = await import('../../src/lib/server/fleet/create.js');
const { housePartyTargets, queueHouseParty } = await import('../../src/lib/server/fleet/house-party.js');
const { db } = await import('../../src/lib/server/db/index.js');
expect(queueCreateServer(input).ok).toBe(true);
expect(queueCreateServer(input).ok).toBe(true);
db().exec(`CREATE TRIGGER reject_destroy BEFORE INSERT ON server_deployments
WHEN NEW.action = 'destroy' AND NEW.server = 't5_zombie_cosmodrome_2'
BEGIN SELECT RAISE(ABORT, 'test failure'); END`);
expect(() => queueHouseParty(housePartyTargets())).toThrow('test failure');
expect(db().prepare("SELECT * FROM server_deployments WHERE action = 'destroy'").all()).toEqual([]);
});
it('waits for queued creations, then for verified emptiness, and preserves keys', async () => {
const { queueCreateServer, processCreations, processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { housePartyTargets, queueHouseParty } = await import('../../src/lib/server/fleet/house-party.js');
const { getServer } = await import('../../src/lib/server/fleet/servers.js');
const { getServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
const { komodoDestroyDeployment, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
expect(queueCreateServer(input).ok).toBe(true);
expect(queueHouseParty(housePartyTargets()).ok).toBe(true);
await processDestructions();
expect(komodoDestroyDeployment).not.toHaveBeenCalled();
await processCreations();
vi.mocked(rconGetPlayerCount).mockResolvedValueOnce(2);
await processDestructions();
vi.mocked(rconGetPlayerCount).mockResolvedValueOnce(null);
await processDestructions();
expect(komodoDestroyDeployment).not.toHaveBeenCalled();
vi.mocked(rconGetPlayerCount).mockResolvedValueOnce(0);
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, kind: 'not-found', message: 'gone' } as never);
await processDestructions();
expect(komodoDestroyDeployment).toHaveBeenCalledWith(serverName);
expect(getServer(serverName)).toBeUndefined();
expect(getServerKey(serverName)).toBe('test-key');
});
});
async function promotedConfig() {
const { createScript, getLatestVersion } = await import('../../src/lib/server/fleet/scripts.js');
const { db } = await import('../../src/lib/server/db/index.js');
for (const artifact of [
{ name: 'baseline', kind: 'baseline' as const, scopeType: 'game' as const, scopeValue: 't5',
content: Buffer.from('set sv_voice "1"\nset sv_maprotation "map zombie_theater"\n') },
{ name: 'map settings', kind: 'config' as const, scopeType: 'map' as const, scopeValue: 'zombie_cosmodrome',
content: Buffer.from('set sv_maprotation "map zombie_moon"\nset g_inactivity "0"\n') },
{ name: 'other map', kind: 'config' as const, scopeType: 'map' as const, scopeValue: 'zombie_theater',
content: Buffer.from('set sv_voice "0"\n') }
]) {
const { script } = createScript(artifact);
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?')
.run(getLatestVersion(script.id)!.id, script.id);
}
}
describe('T6 Zombies provisioning', () => {
it('keeps T6 storage and config overrides separate from T5 settings', async () => {
vi.stubEnv('T6_SERVERS_ROOT', join(fixture.root, 'custom-t6'));
vi.stubEnv('T6_CONFIG_TARGET_FILE', 'custom_zm.cfg');
vi.stubEnv('T6_REL_DEST_SCRIPT', 'custom/scripts/zm');
vi.stubEnv('PLUTAINER_T6_GAMEFILES_VOLUME', 'bo2-files');
const { createScript } = await import('../../src/lib/server/fleet/scripts.js');
const { db } = await import('../../src/lib/server/db/index.js');
const baseline = createScript({ name: 'T6 baseline', kind: 'baseline', scopeType: 'game', scopeValue: 't6', content: Buffer.from('sv_voice 1\n') });
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(baseline.version.id, baseline.script.id);
const { createServer } = await import('../../src/lib/server/fleet/create.js');
const { komodoCreateDeployment } = await import('../../src/lib/server/integrations/komodo.js');
const { destOf, configFileOf } = await import('../../src/lib/server/config.js');
const created = await createServer({ ...input, game: 't6', mapSlug: 'town' });
if (!created.ok) throw new Error(created.message);
const spec = vi.mocked(komodoCreateDeployment).mock.calls[0][0];
expect(spec.volumes).toContain('bo2-files:/home/plutainer/gamefiles:ro');
expect(spec.environment).toContainEqual({ variable: 'PLUTAINER_CONFIG_FILE', value: 'custom_zm.cfg' });
expect(existsSync(join(fixture.root, 'custom-t6', created.name, 'configs/custom_zm.cfg'))).toBe(true);
expect(destOf(created.name)).toBe(`${join(fixture.root, 'custom-t6')}/${created.name}/custom/scripts/zm`);
expect(configFileOf(serverName)).toBe('dedicated_sp.cfg');
expect(destOf(serverName)).toBe(`${fixture.t5}/${serverName}/runtime/plutonium/storage/t5/raw/scripts/sp`);
});
it.each(T6_ZOMBIE_MAPS)('provisions $friendly with the T6 mount, config, and ruleset', async (map) => {
const { createServer } = await import('../../src/lib/server/fleet/create.js');
const { getServer } = await import('../../src/lib/server/fleet/servers.js');
const { komodoCreateDeployment } = await import('../../src/lib/server/integrations/komodo.js');
const name = `t6_zombie_${map.slug}_1`;
expect(await createServer({ ...input, game: 't6', mapSlug: map.slug })).toMatchObject({ ok: true, name });
const spec = vi.mocked(komodoCreateDeployment).mock.calls[0][0];
const env = Object.fromEntries(spec.environment.map(({ variable, value }) => [variable, value]));
expect(env).toMatchObject({ PLUTAINER_GAME: 't6zm', PLUTAINER_CONFIG_FILE: 'dedicated_zm.cfg',
PLUTAINER_RCON_PASSWORD: 'test-rcon', PLUTAINER_MAP_ROTATE: 'false',
PLUTAINER_EXTRA_ARGS: `+exec electric_cherry_map.cfg +set tws_server_id ${name} +map_rotate` });
expect(env.PLUTAINER_SKIP_SEED).toBeUndefined();
expect(spec.volumes).toEqual([`${fixture.t6}/${name}:/home/plutainer/app`, 't6_game_files:/home/plutainer/gamefiles:ro']);
expect(getServer(name)).toMatchObject({ game: 't6', game_id: 't6zm', map: map.canonical,
map_slug: map.slug, config_file: 'dedicated_zm.cfg' });
expect(await readFile(join(fixture.t6, name, 'configs/electric_cherry_map.cfg'), 'utf8'))
.toContain(`set sv_maprotation "execgts ${map.gameSettings} map ${map.canonical}"`);
expect(existsSync(join(fixture.t5, name))).toBe(false);
expect(existsSync(join(fixture.t6, name, 'configs/dedicated_zm.cfg'))).toBe(false);
});
it('persists T6 queued work, reuses its key, and allocates ports across both games', async () => {
const { queueCreateServer, processCreations } = await import('../../src/lib/server/fleet/create.js');
const { saveServerKey, validServerId } = await import('../../src/lib/server/fleet/server-keys.js');
const { closeDb } = await import('../../src/lib/server/db/index.js');
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
const { rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js');
const { getDeployment } = await import('../../src/lib/server/fleet/deployments.js');
saveServerKey('t6_zombie_town_1', 't6-key');
expect(validServerId('t6_zombie_town_20')).toBe(true);
expect(validServerId('t5_zombie_town_1')).toBe(false);
expect(validServerId('t6_zombie_theater_1')).toBe(false);
const t5 = queueCreateServer(input);
const t6 = queueCreateServer({ game: 't6', gameMode: 'zombie', mapSlug: 'town' });
if (!t5.ok || !t6.ok) throw new Error('queue failed');
expect(t6.port).not.toBe(t5.port);
closeDb();
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: true, state: 'running' });
vi.mocked(rconStatusResult).mockImplementation(async (server) => ({ ok: true, endpoint: { host: 'localhost', port: 28960 }, rawReply: '',
status: { players: 0, map: server.startsWith('t6_') ? 'zm_transit' : 'zombie_cosmodrome', gametype: null, roster: [] } }));
await processCreations();
expect(getDeployment(t5.deploymentId)?.deployment).toMatchObject({ state: 'awaiting-health' });
expect(getDeployment(t6.deploymentId)?.deployment).toMatchObject({ state: 'queued' });
await processCreations();
expect(getDeployment(t5.deploymentId)?.deployment).toMatchObject({ state: 'done' });
expect(getDeployment(t6.deploymentId)?.deployment).toMatchObject({ state: 'awaiting-health' });
await processCreations();
expect(getDeployment(t6.deploymentId)?.deployment).toMatchObject({ state: 'done' });
const content = await readFile(join(fixture.t6, t6.name, 'configs/electric_cherry_map.cfg'), 'utf8');
expect(content).toContain('execgts zm_standard_town.cfg map zm_transit');
});
it('installs only matching promoted artifacts into T6 paths and removes only its directory on destruction', async () => {
const { createScript } = await import('../../src/lib/server/fleet/scripts.js');
const { db } = await import('../../src/lib/server/db/index.js');
for (const game of ['t5', 't6']) {
for (const kind of ['script', 'baseline'] as const) {
const { script, version } = createScript({ name: `${game}-${kind}`, kind, scopeType: 'game', scopeValue: game,
targetRelPath: kind === 'script' ? `${game}.gsc` : undefined,
content: Buffer.from(kind === 'script' ? 'main(){}' : 'sv_voice 1\nsv_maprotation "map wrong"\n') });
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(version.id, script.id);
}
}
const { createServer, destroyServer, processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { getServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
const { scanServerDrift } = await import('../../src/lib/server/fleet/drift.js');
await createServer(input);
const t6 = await createServer({ ...input, game: 't6', mapSlug: 'tomb' });
if (!t6.ok) throw new Error(t6.message);
expect(existsSync(join(fixture.t6, t6.name, 'runtime/plutonium/storage/t6/scripts/zm/t6.gsc'))).toBe(true);
expect(existsSync(join(fixture.t6, t6.name, 'runtime/plutonium/storage/t6/scripts/zm/t5.gsc'))).toBe(false);
expect(await readFile(join(fixture.t6, t6.name, 'configs/dedicated_zm.cfg'), 'utf8'))
.toContain('execgts zm_classic_tomb.cfg map zm_tomb');
expect(await scanServerDrift(t6.name)).toEqual([]);
expect(destroyServer(t6.name).ok).toBe(true);
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: false, reason: 'gone' });
await processDestructions();
expect(existsSync(join(fixture.t6, t6.name))).toBe(false);
expect(existsSync(join(fixture.t5, serverName))).toBe(true);
expect(getServerKey(t6.name)).toBe('test-key');
});
it('validates game/map combinations at the API boundary', async () => {
const { POST } = await import('../../src/routes/api/servers/+server.js');
for (const game of ['t7', null, 6]) {
const res = await POST({ request: new Request('http://localhost/api/servers', {
method: 'POST', body: JSON.stringify({ ...input, game })
}) });
expect(res.status).toBe(400);
}
const { queueCreateServer } = await import('../../src/lib/server/fleet/create.js');
expect(queueCreateServer({ ...input, game: 't6' })).toMatchObject({ ok: false, kind: 'unknown-map' });
});
});
describe('server creation map selection', () => {
it('returns a queued creation before provisioning and reserves its name and port', async () => {
const { POST } = await import('../../src/routes/api/servers/+server.js');
const { queueCreateServer, processCreations } = await import('../../src/lib/server/fleet/create.js');
const { getDeployment, interruptDeployments } = await import('../../src/lib/server/fleet/deployments.js');
const { komodoCreateDeployment, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
const { closeDb } = await import('../../src/lib/server/db/index.js');
const response = await POST({ request: new Request('http://localhost/api/servers', {
method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(input)
}) });
expect(response.status).toBe(202);
const body = await response.json();
expect(body).toMatchObject({ name: serverName, deploymentId: expect.any(String) });
expect(komodoCreateDeployment).not.toHaveBeenCalled();
expect(getDeployment(body.deploymentId)?.deployment).toMatchObject({ state: 'queued', expected_map: 'zombie_cosmodrome' });
expect(queueCreateServer({ ...input, expectedName: serverName })).toMatchObject({ ok: false, kind: 'no-slot' });
const second = queueCreateServer(input);
expect(second).toMatchObject({ ok: true, name: 't5_zombie_cosmodrome_2' });
if (second.ok) expect(second.port).not.toBe(body.port);
closeDb();
interruptDeployments();
expect(getDeployment(body.deploymentId)?.deployment).toMatchObject({ state: 'queued' });
await Promise.all([processCreations(), processCreations()]);
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
expect(getDeployment(body.deploymentId)?.deployment).toMatchObject({ state: 'awaiting-health', finished_at: null });
if (second.ok) expect(getDeployment(second.deploymentId)?.deployment).toMatchObject({ state: 'queued' });
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: false, reason: 'starting' });
await processCreations();
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
});
it('completes creation only when Komodo, RCON, and the selected map all pass', async () => {
const { createServer, processCreations } = await import('../../src/lib/server/fleet/create.js');
const { getDeployment, interruptDeployments } = await import('../../src/lib/server/fleet/deployments.js');
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
const { rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js');
const created = await createServer(input);
if (!created.ok) throw new Error('fixture creation failed');
const id = created.deploymentId;
expect(getDeployment(id)?.deployment).toMatchObject({ state: 'awaiting-health', finished_at: null });
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: false, reason: 'starting' });
await processCreations();
expect(rconStatusResult).not.toHaveBeenCalled();
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: true, state: 'running' });
vi.mocked(rconStatusResult).mockResolvedValueOnce({ ok: false, endpoint: null, reason: 'timeout', detail: 'not ready' });
await processCreations();
expect(getDeployment(id)?.deployment).toMatchObject({ state: 'awaiting-health', stage: 'Waiting for RCON or fresh telemetry health: timeout' });
for (const map of [null, 'zombie_moon']) {
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: true, state: 'running' });
if (created.port === null) throw new Error('A provisioned legacy server must have a port.');
vi.mocked(rconStatusResult).mockResolvedValueOnce({ ok: true, endpoint: { host: 'localhost', port: created.port }, rawReply: '', status: { players: 0, map, gametype: null, roster: [] } });
await processCreations();
expect(getDeployment(id)?.deployment).toMatchObject({ state: 'awaiting-health', finished_at: null, stage: expect.stringContaining('expected zombie_cosmodrome') });
}
interruptDeployments();
expect(getDeployment(id)?.deployment).toMatchObject({ state: 'awaiting-health' });
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: true, state: 'running' });
await processCreations();
expect(getDeployment(id)?.deployment).toMatchObject({ state: 'done', finished_at: expect.any(String), stage: 'Verified Komodo healthy, RCON healthy, map zombie_cosmodrome, and Plutonium r5354 (production r5354)' });
});
it('returns a deployment URL identifier immediately and serves its lifecycle detail', async () => {
const { createServer } = await import('../../src/lib/server/fleet/create.js');
const { DELETE } = await import('../../src/routes/api/servers/[name]/+server.js');
const { GET } = await import('../../src/routes/api/deployments/[id]/+server.js');
const { komodoDestroyDeployment } = await import('../../src/lib/server/integrations/komodo.js');
await createServer(input);
const response = await DELETE({ params: { name: serverName } });
expect(response.status).toBe(202);
const body = await response.json();
expect(body).toMatchObject({ ok: true, deploymentId: expect.any(String) });
expect(komodoDestroyDeployment).not.toHaveBeenCalled();
const detail = await (await GET({ params: { id: body.deploymentId } })).json();
expect(detail.deployment).toMatchObject({ server: serverName, action: 'destroy', state: 'waiting' });
expect(detail.events.map((event: { message: string }) => event.message)).toEqual(expect.arrayContaining([
'Server destruction requested', 'Waiting to verify the server is empty'
]));
expect(detail.events.every((event: { id: number }, index: number) => index === 0 || detail.events[index - 1].id < event.id)).toBe(true);
const missing = await DELETE({ params: { name: 'missing-server' } });
expect(missing.status).toBe(404);
});
it('queues destruction, waits for live zero players, and preserves an ordered timeline', async () => {
const { createServer, destroyServer, processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
const { komodoDestroyDeployment, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
const { getDeployment, interruptDeployments } = await import('../../src/lib/server/fleet/deployments.js');
const { closeDb } = await import('../../src/lib/server/db/index.js');
await createServer(input);
const queued = destroyServer(serverName);
expect(queued).toMatchObject({ ok: true, deploymentId: expect.any(String) });
expect(destroyServer(serverName).deploymentId).toBe(queued.deploymentId);
expect(komodoDestroyDeployment).not.toHaveBeenCalled();
vi.mocked(rconGetPlayerCount).mockResolvedValueOnce(3).mockResolvedValueOnce(null).mockRejectedValueOnce(new Error('RCON unavailable'));
for (let i = 0; i < 3; i++) {
await processDestructions();
expect(komodoDestroyDeployment).not.toHaveBeenCalled();
expect(getDeployment(queued.deploymentId!)?.deployment).toMatchObject({ state: 'waiting' });
}
closeDb();
interruptDeployments();
expect(getDeployment(queued.deploymentId!)?.deployment).toMatchObject({ state: 'waiting' });
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: false, reason: 'gone' });
await Promise.all([processDestructions(), processDestructions()]);
expect(komodoDestroyDeployment).toHaveBeenCalledTimes(1);
const detail = getDeployment(queued.deploymentId!);
expect(detail?.deployment).toMatchObject({ state: 'done', server_exists: 0 });
const messages = (detail?.events as { message: string }[]).map(e => e.message);
expect(messages).toEqual(expect.arrayContaining(['Server destruction requested', 'Waiting for server to empty: 3 player(s) online', 'Server is empty (0 players verified)', 'Stopping and removing container', 'Completed']));
expect(messages.indexOf('Server is empty (0 players verified)')).toBeLessThan(messages.indexOf('Stopping and removing container'));
});
it('preserves files and registry when Komodo rejects destruction and exposes failure details', async () => {
const { createServer, destroyServer, processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { komodoDestroyDeployment, komodoDeleteDeployment } = await import('../../src/lib/server/integrations/komodo.js');
const { getDeployment } = await import('../../src/lib/server/fleet/deployments.js');
await createServer(input);
vi.mocked(komodoDestroyDeployment).mockResolvedValueOnce({ ok: false, kind: 'http-error', message: 'denied' });
const queued = destroyServer(serverName);
await processDestructions();
expect(komodoDeleteDeployment).not.toHaveBeenCalled();
expect(existsSync(join(fixture.t5, serverName))).toBe(true);
expect(getDeployment(queued.deploymentId!)?.deployment).toMatchObject({ state: 'failed', server_exists: 1, error: 'DestroyDeployment failed: denied' });
});
it('records in-flight creation and retains both operations after destruction', async () => {
const { createServer, destroyServer, processDestructions, processCreations } = await import('../../src/lib/server/fleet/create.js');
const { listDeployments } = await import('../../src/lib/server/fleet/deployments.js');
const { komodoDeploy, komodoInspectHealthy, komodoDestroyDeployment } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoDeploy).mockImplementationOnce(async () => {
expect(listDeployments()).toContainEqual(expect.objectContaining({ server: serverName, action: 'create', state: 'running', stage: 'Starting container' }));
return { ok: true };
});
expect((await createServer(input)).ok).toBe(true);
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: true, state: 'running' });
await processCreations();
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: false, reason: 'gone' });
vi.mocked(komodoDestroyDeployment).mockImplementationOnce(async () => {
expect(listDeployments()).toContainEqual(expect.objectContaining({ action: 'destroy', state: 'running' }));
return { ok: true };
});
expect((await destroyServer(serverName)).ok).toBe(true);
await processDestructions();
expect(listDeployments()).toEqual(expect.arrayContaining([
expect.objectContaining({ server: serverName, action: 'create', state: 'done', server_exists: 0 }),
expect.objectContaining({ server: serverName, action: 'destroy', state: 'done', server_exists: 0 })
]));
});
it('records failed and unexpectedly interrupted creation attempts', async () => {
const { createServer } = await import('../../src/lib/server/fleet/create.js');
const { listDeployments, beginDeployment, interruptDeployments } = await import('../../src/lib/server/fleet/deployments.js');
const { komodoDeploy } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoDeploy).mockResolvedValueOnce({ ok: false, kind: 'http-error', message: 'test failure' });
expect((await createServer(input)).ok).toBe(false);
expect(listDeployments()).toContainEqual(expect.objectContaining({ state: 'failed', error: 'Deploy: test failure' }));
vi.mocked(komodoDeploy).mockRejectedValueOnce(new Error('connection lost'));
await expect(createServer(input)).rejects.toThrow('connection lost');
expect(listDeployments()).toContainEqual(expect.objectContaining({ state: 'awaiting-health',
stage: 'Verifying existing server after interrupted startup', finished_at: null }));
const id = beginDeployment('interrupted-server', 'destroy');
interruptDeployments();
expect(listDeployments()).toContainEqual(expect.objectContaining({ id, state: 'interrupted', finished_at: expect.any(String) }));
});
it('installs only promoted scope-matching scripts and plugins before first boot', async () => {
const { createScript } = await import('../../src/lib/server/fleet/scripts.js');
const { db } = await import('../../src/lib/server/db/index.js');
const { destOfKind } = await import('../../src/lib/server/config.js');
const { getServerScript } = await import('../../src/lib/server/fleet/inventory.js');
const { createServer } = await import('../../src/lib/server/fleet/create.js');
const { komodoDeploy } = await import('../../src/lib/server/integrations/komodo.js');
const { scanServerDrift } = await import('../../src/lib/server/fleet/drift.js');
const artifacts = [
{ kind: 'script' as const, target: 'zm_spawn_fix.gsc', scopeType: 'game' as const, scopeValue: 't5', promoted: true, expected: true },
{ kind: 'plugin' as const, target: 'plugin.dll', scopeType: 'map' as const, scopeValue: 'zombie_cosmodrome', promoted: true, expected: true },
{ kind: 'script' as const, target: 'other.gsc', scopeType: 'map' as const, scopeValue: 'zombie_theater', promoted: true, expected: false },
{ kind: 'script' as const, target: 'draft.gsc', scopeType: 'game' as const, scopeValue: 't5', promoted: false, expected: false }
].map((a) => {
const { script, version } = createScript({ name: a.target, kind: a.kind, targetRelPath: a.target,
scopeType: a.scopeType, scopeValue: a.scopeValue, content: Buffer.from(a.target) });
if (a.promoted) db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(version.id, script.id);
return { ...a, version };
});
vi.mocked(komodoDeploy).mockImplementationOnce(async () => {
for (const a of artifacts) {
const path = join(destOfKind(serverName, a.kind), a.target);
expect(existsSync(path)).toBe(a.expected);
if (a.expected) {
expect(await readFile(path)).toEqual(a.version.content);
expect(getServerScript(serverName, a.target)).toMatchObject({ script_version_id: a.version.id, source: 'create' });
}
}
return { ok: true };
});
expect(await createServer(input)).toMatchObject({ ok: true });
expect(await scanServerDrift(serverName)).toEqual([]);
});
it('keeps Plutainer defaults without a baseline and overrides the map before rotation', async () => {
const { createServer } = await import('../../src/lib/server/fleet/create.js');
const { komodoCreateDeployment, komodoDeploy } = await import('../../src/lib/server/integrations/komodo.js');
const { getServer } = await import('../../src/lib/server/fleet/servers.js');
const result = await createServer(input);
expect(result).toMatchObject({ ok: true, name: serverName });
expect(existsSync(join(fixture.t5, serverName, 'configs', 'dedicated_sp.cfg'))).toBe(false);
const spec = vi.mocked(komodoCreateDeployment).mock.calls[0][0];
const env = Object.fromEntries(spec.environment.map(({ variable, value }) => [variable, value]));
expect(env.PLUTAINER_MAP_ROTATE).toBe('false');
expect(env.PLUTAINER_EXTRA_ARGS).toBe(`+exec electric_cherry_map.cfg +set tws_server_id ${serverName} +map_rotate`);
expect(await readFile(join(fixture.t5, serverName, 'configs', 'electric_cherry_map.cfg'), 'utf8'))
.toContain('set sv_maprotation "map zombie_cosmodrome"');
expect(env.PLUTAINER_CONFIG_FILE).toBe('dedicated_sp.cfg');
expect(env.PLUTAINER_SKIP_SEED).toBeUndefined();
expect(komodoDeploy).toHaveBeenCalledWith(serverName);
expect(getServer(serverName)?.map).toBe('zombie_cosmodrome');
const { scanServerDrift, runDriftDeployment } = await import('../../src/lib/server/fleet/drift.js');
expect(await scanServerDrift(serverName)).toEqual([]);
const mapPath = join(fixture.t5, serverName, 'configs', 'electric_cherry_map.cfg');
await rm(mapPath);
const repair = await runDriftDeployment({ servers: [serverName] });
expect(repair.totalRestored).toBe(1);
expect(existsSync(mapPath)).toBe(false);
const { activateFiles } = await import('../helpers/staged-files.js');
await activateFiles('campaign', repair.deploymentId!);
expect(await readFile(mapPath, 'utf8')).toContain('set sv_maprotation "map zombie_cosmodrome"');
expect(await scanServerDrift(serverName)).toEqual([]);
});
it('writes promoted settings and selected map before first boot and tracks the file', async () => {
await promotedConfig();
const { createServer } = await import('../../src/lib/server/fleet/create.js');
const { komodoDeploy } = await import('../../src/lib/server/integrations/komodo.js');
const { getServerScript } = await import('../../src/lib/server/fleet/inventory.js');
const { scanServerDrift } = await import('../../src/lib/server/fleet/drift.js');
vi.mocked(komodoDeploy).mockImplementationOnce(async () => {
const config = await readFile(join(fixture.t5, serverName, 'configs', 'dedicated_sp.cfg'), 'utf8');
expect(config).toContain('set sv_maprotation "map zombie_cosmodrome"');
expect(config).toContain('set sv_voice "1"');
expect(config).toContain('set g_inactivity "0"');
expect(config).not.toContain('map zombie_moon');
expect(getServerScript(serverName, 'dedicated_sp.cfg')?.source).toBe('create');
return { ok: true };
});
expect(await createServer(input)).toMatchObject({ ok: true });
expect(await scanServerDrift(serverName)).toEqual([]);
});
it('removes the generated config and inventory if deployment fails', async () => {
await promotedConfig();
const { createScript } = await import('../../src/lib/server/fleet/scripts.js');
const { db } = await import('../../src/lib/server/db/index.js');
const { script, version } = createScript({ name: 'spawn fix', kind: 'script', scopeType: 'game', scopeValue: 't5',
targetRelPath: 'zm_spawn_fix.gsc', content: Buffer.from('main(){}') });
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(version.id, script.id);
const { createServer } = await import('../../src/lib/server/fleet/create.js');
const { komodoDeploy } = await import('../../src/lib/server/integrations/komodo.js');
const { getServer } = await import('../../src/lib/server/fleet/servers.js');
const { getServerScript } = await import('../../src/lib/server/fleet/inventory.js');
vi.mocked(komodoDeploy).mockResolvedValueOnce({ ok: false, kind: 'http-error', message: 'test failure' });
expect(await createServer(input)).toMatchObject({ ok: false, kind: 'komodo' });
expect(getServer(serverName)).toBeUndefined();
expect(getServerScript(serverName, 'dedicated_sp.cfg')).toBeUndefined();
expect(getServerScript(serverName, 'zm_spawn_fix.gsc')).toBeUndefined();
expect(existsSync(join(fixture.t5, serverName))).toBe(false);
});
it('rejects unknown maps before creating a deployment', async () => {
const { createServer } = await import('../../src/lib/server/fleet/create.js');
const { komodoCreateDeployment } = await import('../../src/lib/server/integrations/komodo.js');
expect(await createServer({ ...input, mapSlug: 'unknown' })).toMatchObject({ ok: false, kind: 'unknown-map' });
expect(komodoCreateDeployment).not.toHaveBeenCalled();
});
});
describe('persistent server keys', () => {
it('saves a key, retains it through destruction, and reuses it for the exact lobby', async () => {
const { createServer, destroyServer, processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { getServerKey, listServerKeys } = await import('../../src/lib/server/fleet/server-keys.js');
const { komodoCreateDeployment, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
const name = 't5_zombie_cosmodrome_2';
expect(await createServer(input)).toMatchObject({ ok: true, name: serverName });
expect(await createServer(input)).toMatchObject({ ok: true, name });
expect(getServerKey(name)).toBe('test-key');
expect(await destroyServer(name)).toMatchObject({ ok: true });
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, reason: 'gone' });
await processDestructions();
expect(listServerKeys()).toContainEqual(expect.objectContaining({ server_id: name, deployed: 0 }));
expect(JSON.stringify(listServerKeys())).not.toContain('test-key');
expect(await createServer({ gameMode: 'zombie', mapSlug: 'cosmodrome', expectedName: name })).toMatchObject({ ok: true, name });
const spec = vi.mocked(komodoCreateDeployment).mock.calls.at(-1)![0];
expect(spec.environment).toContainEqual({ variable: 'PLUTO_SERVER_KEY', value: 'test-key' });
expect(await createServer({ gameMode: 'zombie', mapSlug: 'cosmodrome' })).toMatchObject({ ok: false, kind: 'bad-request' });
});
it('updates and removes keys independently of deployments and survives reopening the database', async () => {
const { saveServerKey, getServerKey, removeServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
const { closeDb } = await import('../../src/lib/server/db/index.js');
saveServerKey(serverName, ' first-key ');
closeDb();
expect(getServerKey(serverName)).toBe('first-key');
saveServerKey(serverName, 'replacement');
expect(getServerKey(serverName)).toBe('replacement');
expect(removeServerKey(serverName)).toBe(true);
expect(getServerKey(serverName)).toBeUndefined();
expect(() => saveServerKey('t5_zombie_unknown_1', 'key')).toThrow();
expect(() => saveServerKey(serverName, ' ')).toThrow();
expect(() => saveServerKey(serverName, 'two words')).toThrow();
});
it('fills the lowest gap and rejects stale previews before saving a key or deploying', async () => {
const { createServer, destroyServer, processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { getServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
const { komodoCreateDeployment, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
expect(await createServer(input)).toMatchObject({ ok: true, name: serverName });
expect(await createServer(input)).toMatchObject({ ok: true, name: 't5_zombie_cosmodrome_2' });
await destroyServer(serverName);
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, reason: 'gone' });
await processDestructions();
expect(await createServer({ ...input, plutoKey: 'wrong-key', expectedName: 't5_zombie_cosmodrome_3' }))
.toMatchObject({ ok: false, kind: 'no-slot' });
expect(getServerKey(serverName)).toBe('test-key');
expect(komodoCreateDeployment).toHaveBeenCalledTimes(2);
expect(await createServer({ gameMode: 'zombie', mapSlug: 'cosmodrome', expectedName: serverName }))
.toMatchObject({ ok: true, name: serverName });
});
});
describe('cancelling submitted creation and destruction', () => {
it('retains a remotely created identity after cancellation without starting its container', async () => {
const { queueCreateServer, processCreations, nextAvailablePort, nextLobbyNumber } = await import('../../src/lib/server/provisioning/instances.js');
const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
const { getDeployment } = await import('../../src/lib/server/provisioning/history.js');
const { getServer } = await import('../../src/lib/server/fleet/servers.js');
const { komodoCreateDeployment, komodoDeploy } = await import('../../src/lib/server/integrations/komodo.js');
const queued = queueCreateServer(input); if (!queued.ok) throw new Error(queued.message);
let release!: (value: {ok:true}) => void;
vi.mocked(komodoCreateDeployment).mockImplementationOnce(() => new Promise(resolve => {release=resolve;}));
const work = processCreations();
await vi.waitFor(() => expect(release).toBeTypeOf('function'));
expect(cancelProvisioningJob(queued.deploymentId!)).toBe(true);
expect(nextAvailablePort()).not.toBe(queued.port);
expect(nextLobbyNumber('t5_zombie_cosmodrome')).toBe(2);
release({ok:true}); await work;
expect(getServer(queued.name)).toBeTruthy();
expect(komodoDeploy).not.toHaveBeenCalled();
expect(getDeployment(queued.deploymentId!)?.deployment.state).toBe('cancelled');
});
it('does not delete deployment metadata or files after cancelling container teardown', async () => {
const { createServer, destroyServer, processDestructions } = await import('../../src/lib/server/provisioning/instances.js');
const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
const { getDeployment } = await import('../../src/lib/server/provisioning/history.js');
const { getServer } = await import('../../src/lib/server/fleet/servers.js');
const { komodoDestroyDeployment, komodoDeleteDeployment } = await import('../../src/lib/server/integrations/komodo.js');
await createServer(input);
let release!: (value: {ok:true}) => void;
vi.mocked(komodoDestroyDeployment).mockImplementationOnce(() => new Promise(resolve => {release=resolve;}));
const queued=destroyServer(serverName); const work=processDestructions();
await vi.waitFor(() => expect(release).toBeTypeOf('function'));
expect(cancelProvisioningJob(queued.deploymentId!)).toBe(true);
release({ok:true}); await work;
expect(komodoDeleteDeployment).not.toHaveBeenCalled();
expect(getServer(serverName)).toBeTruthy();
expect(existsSync(join(fixture.t5,serverName))).toBe(true);
expect(getDeployment(queued.deploymentId!)?.deployment.state).toBe('cancelled');
});
});
-403
View File
@@ -1,403 +0,0 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { existsSync } from 'node:fs';
import { join } from 'node:path';
import { newFixture, type Fixture } from '../helpers/fixture.js';
vi.mock('../../src/lib/server/integrations/komodo.js', () => ({
komodoCreateDeployment: vi.fn(),
komodoDeploy: vi.fn(),
komodoDestroyDeployment: vi.fn(),
komodoDeleteDeployment: vi.fn(),
komodoInspectHealthy: vi.fn(),
komodoEnsureTag: vi.fn(),
komodoUpdateTagsOnResource: vi.fn()
}));
vi.mock('../../src/lib/server/integrations/rcon.js', () => ({
rconGetPlayerCount: vi.fn(),
rconPlutoniumVersion: vi.fn(),
rconStatusResult: vi.fn()
}));
vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}) }));
let fixture: Fixture;
const input = { gameMode: 'zombie' as const, mapSlug: 'theater', plutoKey: 'test-key' };
function status(server: string, map = server.includes('_moon_') ? 'zombie_moon' : 'zombie_theater') {
return { ok: true as const, endpoint: { host: 'localhost', port: 28960 }, rawReply: '',
status: { players: 0, map, gametype: null, roster: [] } };
}
beforeEach(async () => {
vi.resetModules(); vi.resetAllMocks();
fixture = await newFixture([]);
vi.stubGlobal('fetch', vi.fn(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354, files: [] })));
vi.stubEnv('KOMODO_ADDRESS', 'http://komodo.test');
vi.stubEnv('KOMODO_API_KEY', 'test-key');
vi.stubEnv('KOMODO_API_SECRET', 'test-secret');
vi.stubEnv('KOMODO_TARGET_SERVER', 'test-host');
vi.stubEnv('RCON_PASSWORD', 'test-password');
vi.stubEnv('SERVER_PORT_MIN', '28960');
vi.stubEnv('SERVER_PORT_MAX', '28980');
vi.stubEnv('DEPLOYMENT_CREATE_CONCURRENCY', '1');
const komodo = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodo.komodoCreateDeployment).mockResolvedValue({ ok: true });
vi.mocked(komodo.komodoDeploy).mockResolvedValue({ ok: true });
vi.mocked(komodo.komodoDestroyDeployment).mockResolvedValue({ ok: true });
vi.mocked(komodo.komodoDeleteDeployment).mockResolvedValue({ ok: true });
vi.mocked(komodo.komodoInspectHealthy).mockResolvedValue({ ok: true, state: 'running' });
vi.mocked(komodo.komodoEnsureTag).mockImplementation(async name => ({ ok: true, id: name }));
vi.mocked(komodo.komodoUpdateTagsOnResource).mockResolvedValue({ ok: true });
const rcon = await import('../../src/lib/server/integrations/rcon.js');
vi.mocked(rcon.rconGetPlayerCount).mockResolvedValue(0);
vi.mocked(rcon.rconPlutoniumVersion).mockImplementation(async () => {
// Version evidence must be observed strictly after the launch timestamp.
await new Promise(resolve => setTimeout(resolve, 2));
return { revision: 5354, error: null };
});
vi.mocked(rcon.rconStatusResult).mockImplementation(async server => status(server));
const { kick } = await import('../../src/lib/server/pipeline/worker.js');
vi.mocked(kick).mockResolvedValue(undefined);
});
afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); vi.unstubAllGlobals(); vi.restoreAllMocks(); });
async function queueServers(count: number) {
const { queueCreateServer } = await import('../../src/lib/server/fleet/create.js');
return Array.from({ length: count }, () => {
const result = queueCreateServer(input);
if (!result.ok) throw new Error(result.message);
return result;
});
}
async function states() {
const { db } = await import('../../src/lib/server/db/index.js');
return (db().prepare("SELECT state FROM server_deployments WHERE action = 'create' ORDER BY rowid").all() as { state: string }[]).map(row => row.state);
}
describe('shared creation queue', () => {
it('rechecks a lowered limit after awaiting a startup in the same pass', async () => {
await queueServers(3);
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
saveFleetSettings({ ...fleetSettings(), createConcurrency: 3 });
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
const { komodoCreateDeployment } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoCreateDeployment).mockImplementationOnce(async () => {
saveFleetSettings({ ...fleetSettings(), createConcurrency: 1 });
return { ok: true };
});
await processCreations();
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
expect(await states()).toEqual(['awaiting-health', 'queued', 'queued']);
});
it('applies live increases and decreases while preserving active startups', async () => {
const jobs = await queueServers(4);
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
const { komodoInspectHealthy, komodoCreateDeployment } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, reason: 'still starting' });
await processCreations();
saveFleetSettings({ ...fleetSettings(), createConcurrency: 3 });
await processCreations();
expect(await states()).toEqual(['awaiting-health', 'awaiting-health', 'awaiting-health', 'queued']);
saveFleetSettings({ ...fleetSettings(), createConcurrency: 1 });
await processCreations();
expect(komodoCreateDeployment).toHaveBeenCalledTimes(3);
expect(await states()).toEqual(['awaiting-health', 'awaiting-health', 'awaiting-health', 'queued']);
vi.mocked(komodoInspectHealthy).mockImplementation(async server => server === jobs[2].name
? { ok: false, reason: 'still starting' } : { ok: true, state: 'running' });
await processCreations();
expect(await states()).toEqual(['done', 'done', 'awaiting-health', 'queued']);
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: true, state: 'running' });
await processCreations();
expect(await states()).toEqual(['done', 'done', 'done', 'awaiting-health']);
});
it('starts only one server per pass and holds its slot until a later health check', async () => {
const jobs = await queueServers(3);
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
const { komodoCreateDeployment, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
await processCreations();
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
expect(komodoInspectHealthy).not.toHaveBeenCalled();
expect(await states()).toEqual(['awaiting-health', 'queued', 'queued']);
await processCreations();
expect(komodoInspectHealthy).toHaveBeenCalledWith(jobs[0].name);
expect(komodoCreateDeployment).toHaveBeenCalledTimes(2);
expect(await states()).toEqual(['done', 'awaiting-health', 'queued']);
expect(new Set(jobs.map(job => job.port)).size).toBe(3);
});
it('keeps later boots queued through container, RCON, and expected-map failures', async () => {
const jobs = await queueServers(2);
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
const { komodoCreateDeployment, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
const { rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js');
const { getDeployment } = await import('../../src/lib/server/fleet/deployments.js');
await processCreations();
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: false, reason: 'starting' });
await processCreations();
expect(rconStatusResult).not.toHaveBeenCalled();
expect(getDeployment(jobs[0].deploymentId)?.deployment).toMatchObject({ stage: 'Waiting for Komodo health: starting' });
vi.mocked(rconStatusResult).mockResolvedValueOnce({ ok: false, reason: 'timeout', endpoint: null, detail: 'booting' });
await processCreations();
expect(getDeployment(jobs[0].deploymentId)?.deployment).toMatchObject({ stage: 'Waiting for RCON or fresh telemetry health: timeout' });
vi.mocked(rconStatusResult).mockResolvedValueOnce(status(jobs[0].name, 'zombie_moon'));
await processCreations();
expect(getDeployment(jobs[0].deploymentId)?.deployment).toMatchObject({ stage: expect.stringContaining('expected zombie_theater') });
vi.mocked(komodoInspectHealthy).mockRejectedValueOnce(new Error('Temporary connection error'));
await processCreations();
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
expect(await states()).toEqual(['awaiting-health', 'queued']);
await processCreations();
expect(komodoCreateDeployment).toHaveBeenCalledTimes(2);
expect(await states()).toEqual(['done', 'awaiting-health']);
});
it('shares FIFO capacity across collections and direct callers, including equal timestamps', async () => {
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
const { previewCollection, queueCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
const { createServer, processCreations } = await import('../../src/lib/server/fleet/create.js');
const { db } = await import('../../src/lib/server/db/index.js');
const { komodoCreateDeployment } = await import('../../src/lib/server/integrations/komodo.js');
const collections = ['theater', 'moon'].map(mapSlug => {
const blueprint = saveBlueprint({ name: mapSlug, entries: [{ game: 't5', mapSlug, count: 2 }] });
const preview = previewCollection(blueprint.id);
return queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name, 'test-key'])) });
});
db().prepare("UPDATE server_deployments SET started_at = '2026-01-01T00:00:00.000Z'").run();
const direct = await createServer(input);
if (!direct.ok) throw new Error(direct.message);
expect(await states()).toEqual(['awaiting-health', 'queued', 'queued', 'queued', 'queued']);
const second = getCollection(collections[1].collectionId!)!;
expect(second.collection).toMatchObject({ queued: 2, provisioning: 0, verifying: 0 });
expect(second.queue).toMatchObject({ concurrency: 1, queued: 4, active: [{ server: 't5_zombie_theater_1' }] });
expect(second.servers.map(server => server.queuePosition)).toEqual([2, 3]);
for (let pass = 0; pass < 5; pass++) await processCreations();
expect(vi.mocked(komodoCreateDeployment).mock.calls.map(([spec]) => spec.name)).toEqual([
't5_zombie_theater_1', 't5_zombie_theater_2', 't5_zombie_moon_1', 't5_zombie_moon_2', direct.name
]);
expect(await states()).toEqual(['done', 'done', 'done', 'done', 'done']);
});
it('coalesces overlapping worker passes and direct calls while provisioning is pending', async () => {
await queueServers(3);
const { processCreations, createServer } = await import('../../src/lib/server/fleet/create.js');
const { komodoCreateDeployment } = await import('../../src/lib/server/integrations/komodo.js');
let release!: () => void;
let entered!: () => void;
const blocked = new Promise<void>(resolve => { release = resolve; });
const started = new Promise<void>(resolve => { entered = resolve; });
vi.mocked(komodoCreateDeployment).mockImplementationOnce(async () => { entered(); await blocked; return { ok: true }; });
const first = processCreations();
await started;
const second = processCreations();
const direct = createServer(input);
expect(await states()).toEqual(['running', 'queued', 'queued', 'queued']);
release();
await Promise.all([first, second, direct]);
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
expect(await states()).toEqual(['awaiting-health', 'queued', 'queued', 'queued']);
});
it('honors a higher limit without refilling its pass budget after immediate failures', async () => {
vi.stubEnv('DEPLOYMENT_CREATE_CONCURRENCY', '2');
await queueServers(5);
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
const { komodoCreateDeployment } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoCreateDeployment).mockResolvedValue({ ok: false, kind: 'http-error', message: 'Host unavailable' });
await processCreations();
expect(komodoCreateDeployment).toHaveBeenCalledTimes(2);
expect(await states()).toEqual(['failed', 'failed', 'queued', 'queued', 'queued']);
vi.mocked(komodoCreateDeployment).mockResolvedValue({ ok: true });
await processCreations();
expect(komodoCreateDeployment).toHaveBeenCalledTimes(4);
expect(await states()).toEqual(['failed', 'failed', 'awaiting-health', 'awaiting-health', 'queued']);
});
it('does not provision a selected job that was cancelled during an earlier startup', async () => {
vi.stubEnv('DEPLOYMENT_CREATE_CONCURRENCY', '2');
const jobs = await queueServers(3);
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
const { finishDeployment } = await import('../../src/lib/server/fleet/deployments.js');
const { komodoCreateDeployment } = await import('../../src/lib/server/integrations/komodo.js');
let release!: () => void;
let entered!: () => void;
const blocked = new Promise<void>(resolve => { release = resolve; });
const started = new Promise<void>(resolve => { entered = resolve; });
vi.mocked(komodoCreateDeployment).mockImplementationOnce(async () => { entered(); await blocked; return { ok: true }; });
const pass = processCreations();
await started;
finishDeployment(jobs[1].deploymentId, { ok: false, message: 'Cancelled before startup' });
release();
await pass;
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
expect(await states()).toEqual(['awaiting-health', 'failed', 'queued']);
});
it('rechecks capacity when an existing server resumes verification during provisioning', async () => {
vi.stubEnv('DEPLOYMENT_CREATE_CONCURRENCY', '2');
const { createServer, processCreations } = await import('../../src/lib/server/fleet/create.js');
const { finishDeployment, resumeVerification } = await import('../../src/lib/server/fleet/deployments.js');
const { komodoCreateDeployment } = await import('../../src/lib/server/integrations/komodo.js');
const old = await createServer(input);
if (!old.ok) throw new Error(old.message);
finishDeployment(old.deploymentId, { ok: false, message: 'Needs verification' });
await queueServers(2);
let release!: () => void;
let entered!: () => void;
const blocked = new Promise<void>(resolve => { release = resolve; });
const started = new Promise<void>(resolve => { entered = resolve; });
vi.mocked(komodoCreateDeployment).mockImplementationOnce(async () => { entered(); await blocked; return { ok: true }; });
const pass = processCreations();
await started;
expect(resumeVerification(old.deploymentId)).toBe(true);
release();
await pass;
expect(komodoCreateDeployment).toHaveBeenCalledTimes(2);
expect(await states()).toEqual(['awaiting-health', 'awaiting-health', 'queued']);
});
it('counts already verifying servers against a higher concurrency limit', async () => {
vi.stubEnv('DEPLOYMENT_CREATE_CONCURRENCY', '2');
const jobs = await queueServers(4);
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
const { komodoCreateDeployment, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
await processCreations();
vi.mocked(komodoInspectHealthy).mockImplementation(async server => server === jobs[0].name
? { ok: false, reason: 'still booting' } : { ok: true, state: 'running' });
await processCreations();
expect(komodoCreateDeployment).toHaveBeenCalledTimes(3);
expect(await states()).toEqual(['awaiting-health', 'done', 'awaiting-health', 'queued']);
});
it('retains durable reservations and the startup gate across an application restart', async () => {
const jobs = await queueServers(3);
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
const { komodoCreateDeployment, komodoDeploy, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
await processCreations();
db().prepare("UPDATE server_deployments SET state = 'running' WHERE id = ?").run(jobs[0].deploymentId);
closeDb();
vi.resetModules();
const { interruptDeployments } = await import('../../src/lib/server/fleet/deployments.js');
const { processCreations: resumed, nextAvailablePort } = await import('../../src/lib/server/fleet/create.js');
interruptDeployments();
expect(await states()).toEqual(['interrupted', 'queued', 'queued']);
expect(nextAvailablePort()).toBe(28963);
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: false, reason: 'still booting' });
await resumed();
expect(await states()).toEqual(['awaiting-health', 'queued', 'queued']);
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
expect(komodoDeploy).toHaveBeenCalledTimes(1);
await resumed();
expect(await states()).toEqual(['done', 'awaiting-health', 'queued']);
expect(komodoCreateDeployment).toHaveBeenCalledTimes(2);
});
it('keeps an ambiguous startup response in verification without replaying its deployment', async () => {
await queueServers(2);
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
const { komodoDeploy, komodoCreateDeployment, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoDeploy).mockRejectedValueOnce(new Error('Response lost after starting'));
const log = vi.spyOn(console, 'error').mockImplementation(() => {});
await processCreations();
expect(await states()).toEqual(['awaiting-health', 'queued']);
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: false, reason: 'still booting' });
await processCreations();
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
expect(komodoDeploy).toHaveBeenCalledTimes(1);
expect(log).toHaveBeenCalled();
await processCreations();
expect(await states()).toEqual(['done', 'awaiting-health']);
});
it.each([
{ ok: false as const, kind: 'unreachable' as const, message: 'Request timed out after Komodo accepted it' },
{ ok: false as const, kind: 'http-error' as const, httpStatus: 502, message: 'Gateway lost the startup response' }
])('holds the slot and files for an uncertain structured response: $kind', async failure => {
const jobs = await queueServers(2);
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
const { getServer } = await import('../../src/lib/server/fleet/servers.js');
const { komodoDeploy, komodoDeleteDeployment, komodoCreateDeployment, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoDeploy).mockResolvedValueOnce(failure);
await processCreations();
expect(await states()).toEqual(['awaiting-health', 'queued']);
expect(getServer(jobs[0].name)?.game_port).toBe(jobs[0].port);
expect(existsSync(join(fixture.t5, jobs[0].name, 'configs/electric_cherry_map.cfg'))).toBe(true);
expect(komodoDeleteDeployment).not.toHaveBeenCalled();
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: false, reason: 'still booting' });
await processCreations();
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
expect(komodoDeploy).toHaveBeenCalledTimes(1);
await processCreations();
expect(await states()).toEqual(['done', 'awaiting-health']);
});
it('releases capacity on a confirmed deployment rejection', async () => {
const jobs = await queueServers(2);
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
const { getServer } = await import('../../src/lib/server/fleet/servers.js');
const { komodoDeploy, komodoCreateDeployment } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoDeploy).mockResolvedValueOnce({ ok: false, kind: 'http-error', httpStatus: 403, message: 'Access denied' });
await processCreations();
expect(await states()).toEqual(['failed', 'queued']);
expect(getServer(jobs[0].name)).toBeUndefined();
expect(existsSync(join(fixture.t5, jobs[0].name))).toBe(false);
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
await processCreations();
expect(await states()).toEqual(['failed', 'awaiting-health']);
});
it('retains an interrupted startup slot while a newer destruction is still waiting', async () => {
const jobs = await queueServers(2);
const { processCreations, destroyServer } = await import('../../src/lib/server/fleet/create.js');
const { interruptDeployments } = await import('../../src/lib/server/fleet/deployments.js');
const { db } = await import('../../src/lib/server/db/index.js');
const { komodoCreateDeployment, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
await processCreations();
expect(destroyServer(jobs[0].name)).toMatchObject({ ok: true });
db().prepare("UPDATE server_deployments SET state = 'running' WHERE id = ?").run(jobs[0].deploymentId);
interruptDeployments();
expect(await states()).toEqual(['interrupted', 'queued']);
await processCreations();
expect(await states()).toEqual(['awaiting-health', 'queued']);
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
expect(komodoInspectHealthy).not.toHaveBeenCalled();
});
it('releases a startup slot after its server is removed', async () => {
const jobs = await queueServers(2);
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
const { db } = await import('../../src/lib/server/db/index.js');
await processCreations();
db().prepare('DELETE FROM servers WHERE name = ?').run(jobs[0].name);
await processCreations();
expect(await states()).toEqual(['cancelled', 'awaiting-health']);
});
it('never verifies a replacement registered under the same server name', async () => {
const jobs = await queueServers(2);
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
const { db } = await import('../../src/lib/server/db/index.js');
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
await processCreations();
db().prepare('UPDATE servers SET game_port = ? WHERE name = ?').run(28980, jobs[0].name);
await processCreations();
expect(komodoInspectHealthy).not.toHaveBeenCalledWith(jobs[0].name);
expect(await states()).toEqual(['failed', 'awaiting-health']);
});
});
describe('creation concurrency configuration', () => {
it('defaults to one startup', async () => {
vi.stubEnv('DEPLOYMENT_CREATE_CONCURRENCY', undefined);
const { fleetSettings } = await import('../../src/lib/server/settings.js');
expect(fleetSettings().createConcurrency).toBe(1);
});
it.each(['0', '-1', '1.5', 'invalid', 'Infinity', '9007199254740992'])('rejects invalid limit %s', async value => {
vi.stubEnv('DEPLOYMENT_CREATE_CONCURRENCY', value);
const { fleetSettings } = await import('../../src/lib/server/settings.js');
expect(() => fleetSettings()).toThrow(/Concurrent server startups/);
vi.stubEnv('DEPLOYMENT_CREATE_CONCURRENCY', '1');
});
});
+1 -1
View File
@@ -14,7 +14,7 @@ const path = (server: string) => join(fixture.t5, server, 'runtime/plutonium/sto
beforeEach(async () => { beforeEach(async () => {
vi.resetModules(); vi.resetModules();
fixture = await newFixture(names); fixture = await newFixture(names);
vi.stubEnv('KOMODO_ADDRESS', ''); vi.stubEnv('RCON_PASSWORD', ''); vi.stubEnv('RCON_PASSWORD', '');
const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js'); const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js');
await seedServerRegistry(); await seedServerRegistry();
}); });
+5 -405
View File
@@ -1,18 +1,10 @@
import { prepareNativeHost } from '../helpers/native-host.js';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { readFile } from 'node:fs/promises'; import { readFile } from 'node:fs/promises';
import { join } from 'node:path'; import { join } from 'node:path';
import { newFixture, type Fixture } from '../helpers/fixture.js'; import { newFixture, type Fixture } from '../helpers/fixture.js';
import type { CollectionPreview } from '../../src/lib/deployment-plan.js'; import type { CollectionPreview } from '../../src/lib/deployment-plan.js';
vi.mock('../../src/lib/server/integrations/komodo.js', () => ({
komodoCreateDeployment: vi.fn(async () => ({ ok: true })),
komodoDeploy: vi.fn(async () => ({ ok: true })),
komodoDestroyDeployment: vi.fn(async () => ({ ok: true })),
komodoDeleteDeployment: vi.fn(async () => ({ ok: true })),
komodoInspectHealthy: vi.fn(async () => ({ ok: true, state: 'running' })),
komodoEnsureTag: vi.fn(async (name: string) => ({ ok: true, id: name })),
komodoUpdateTagsOnResource: vi.fn(async () => ({ ok: true }))
}));
vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ vi.mock('../../src/lib/server/integrations/rcon.js', () => ({
rconGetPlayerCount: vi.fn(async () => 0), rconGetPlayerCount: vi.fn(async () => 0),
rconPlutoniumVersion: vi.fn(async () => { rconPlutoniumVersion: vi.fn(async () => {
@@ -34,14 +26,10 @@ const entries = [
beforeEach(async () => { beforeEach(async () => {
vi.resetModules(); vi.clearAllMocks(); vi.resetModules(); vi.clearAllMocks();
fixture = await newFixture([]); fixture = await newFixture([]);
await prepareNativeHost();
vi.stubGlobal('fetch', vi.fn(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354, files: [] }))); vi.stubGlobal('fetch', vi.fn(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354, files: [] })));
vi.stubEnv('KOMODO_ADDRESS', 'http://komodo.test');
vi.stubEnv('KOMODO_API_KEY', 'private-api-key');
vi.stubEnv('KOMODO_API_SECRET', 'private-api-secret');
vi.stubEnv('KOMODO_TARGET_SERVER', 'test-host');
vi.stubEnv('RCON_PASSWORD', 'private-rcon-password'); vi.stubEnv('RCON_PASSWORD', 'private-rcon-password');
vi.stubEnv('SERVER_PORT_MIN', '28960');
vi.stubEnv('SERVER_PORT_MAX', '28980');
vi.stubEnv('DEPLOYMENT_CREATE_CONCURRENCY', '1'); vi.stubEnv('DEPLOYMENT_CREATE_CONCURRENCY', '1');
}); });
afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); vi.unstubAllGlobals(); }); afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); vi.unstubAllGlobals(); });
@@ -67,26 +55,6 @@ describe('multi-server blueprints', () => {
expect(getCollection(result.collectionId!)?.queue.concurrency).toBe(3); expect(getCollection(result.collectionId!)?.queue.concurrency).toBe(3);
}); });
it('saves a mixed-game lineup and previews unique slots and global ports without reserving them', async () => {
const { blueprint, preview } = await previewLineup();
const { db } = await import('../../src/lib/server/db/index.js');
const { previewCreation } = await import('../../src/lib/server/fleet/create.js');
expect(blueprint.totalServers).toBe(5);
expect(blueprint.entries.map(entry => entry.count)).toEqual([2, 1, 2]);
expect(blueprint).not.toHaveProperty('recipe');
const stored = db().prepare('SELECT recipe FROM server_blueprints WHERE id = ?').get(blueprint.id) as { recipe: string };
expect(JSON.parse(stored.recipe)).toEqual({ format: 3, entries });
expect(preview.ready).toBe(true);
expect(preview.launchConcurrency).toBe(1);
expect(preview.servers.map(server => server.name)).toEqual(['t5_zombie_theater_1', 't5_zombie_theater_2', 't5_zombie_moon_1', 't6_zombie_town_1', 't6_zombie_town_2']);
expect(preview.servers.map(server => server.port)).toEqual([28960, 28961, 28962, 28963, 28964]);
expect(db().prepare('SELECT * FROM server_creation_jobs').all()).toEqual([]);
expect(db().prepare('SELECT * FROM server_keys').all()).toEqual([]);
expect(JSON.stringify(preview)).not.toContain('private-');
expect(preview.entries[0].recipe.files[0]).not.toHaveProperty('content');
expect(() => previewCreation({ blueprintId: blueprint.id })).toThrow(/multiple servers/);
});
it('reads a legacy one-server blueprint as a map selection and saves edits without snapshots', async () => { it('reads a legacy one-server blueprint as a map selection and saves edits without snapshots', async () => {
const { captureRecipe, listBlueprints, updateBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js'); const { captureRecipe, listBlueprints, updateBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
const { createScript, uploadNewVersion } = await import('../../src/lib/server/fleet/scripts.js'); const { createScript, uploadNewVersion } = await import('../../src/lib/server/fleet/scripts.js');
@@ -112,149 +80,15 @@ describe('multi-server blueprints', () => {
expect(listBlueprints()).toHaveLength(1); expect(listBlueprints()).toHaveLength(1);
}); });
it('uses current promotions for legacy collection blueprints while retaining counts and exact T6 variants', async () => {
const { captureRecipe, listBlueprints } = await import('../../src/lib/server/fleet/launch-recipes.js');
const { createScript, uploadNewVersion } = await import('../../src/lib/server/fleet/scripts.js');
const { previewCollection, queueCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
const artifact = createScript({ name: 'Legacy telemetry', targetRelPath: 'telemetry.gsc', scopeType: 'game', scopeValue: 't5', content: Buffer.from('legacy telemetry') });
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(artifact.version.id, artifact.script.id);
const legacyEntries = entries.map(({ count, game, mapSlug }) => ({ count, recipe: captureRecipe(game as 't5' | 't6', mapSlug) }));
db().prepare('INSERT INTO server_blueprints (id,name,created_at,recipe) VALUES (?, ?, ?, ?)').run('legacy-lineup', 'Legacy lineup', new Date().toISOString(), JSON.stringify({ format: 2, entries: legacyEntries }));
const current = uploadNewVersion(artifact.script.id, Buffer.from('current telemetry'));
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(current.id, artifact.script.id);
closeDb();
expect(listBlueprints()[0].entries.map(({ mapName: _mapName, ...entry }) => entry)).toEqual(entries);
const preview = previewCollection('legacy-lineup');
expect(preview.entries[0].recipe.files).toContainEqual(expect.objectContaining({ path: 'telemetry.gsc', versionId: current.id }));
expect(preview.entries[2].recipe.mapSlug).toBe('town');
const queued = queueCollection({ previewId: preview.id, keys: keysFor(preview) });
for (let pass = 0; pass <= preview.servers.length; pass++) await processCreations();
expect(getCollection(queued.collectionId!)?.collection).toMatchObject({ total: 5, done: 5 });
for (const name of ['t5_zombie_theater_1', 't5_zombie_theater_2', 't5_zombie_moon_1']) {
expect(await readFile(join(fixture.t5, name, 'runtime/plutonium/storage/t5/raw/scripts/sp/telemetry.gsc'), 'utf8')).toBe('current telemetry');
}
expect(await readFile(join(fixture.t6, 't6_zombie_town_2', 'configs/electric_cherry_map.cfg'), 'utf8')).toContain('execgts zm_standard_town.cfg map zm_transit');
});
it.each([[], [{ game: 't5', mapSlug: 'theater', count: 0 }], [{ game: 't5', mapSlug: 'theater', count: 21 }], it.each([[], [{ game: 't5', mapSlug: 'theater', count: 0 }], [{ game: 't5', mapSlug: 'theater', count: 21 }],
[{ game: 't5', mapSlug: 'theater', count: 1.5 }], [{ game: 't5', mapSlug: 'theater', count: '2' }], [{ game: 't5', mapSlug: 'theater', count: 1.5 }], [{ game: 't5', mapSlug: 'theater', count: '2' }],
[{ game: 't5', mapSlug: 'town', count: 1 }], [entries[0], entries[0]]].map(input => ({ input })))('rejects an invalid lineup: $input', async ({ input }) => { [{ game: 't5', mapSlug: 'town', count: 1 }], [entries[0], entries[0]]].map(input => ({ input })))('rejects an invalid lineup: $input', async ({ input }) => {
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js'); const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
expect(() => saveBlueprint({ name: 'Invalid', entries: input })).toThrow(); expect(() => saveBlueprint({ name: 'Invalid', entries: input })).toThrow();
}); });
it('reports insufficient capacity without preventing blueprint creation', async () => {
vi.stubEnv('SERVER_PORT_MAX', '28961');
const { blueprint, preview } = await previewLineup();
const { queueCollection } = await import('../../src/lib/server/fleet/collections.js');
const { db } = await import('../../src/lib/server/db/index.js');
expect(blueprint.totalServers).toBe(5);
expect(preview.ready).toBe(false);
expect(preview.servers.filter(server => server.port === null)).toHaveLength(3);
expect(() => queueCollection({ previewId: preview.id, keys: keysFor(preview) })).toThrow(/not enough free game ports/);
expect(db().prepare('SELECT * FROM server_launch_collections').all()).toEqual([]);
});
}); });
describe('atomic collection launches', () => { describe('atomic collection launches', () => {
it('fills the target shortfall and counts registered/in-flight identities only once', async () => {
const { createServer } = await import('../../src/lib/server/fleet/create.js');
for (let i = 0; i < 2; i++) expect((await createServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'existing-key' })).ok).toBe(true);
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
const { previewCollection, queueCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
const blueprint = saveBlueprint({ name: 'Friday targets', entries: [
{ game: 't5', mapSlug: 'theater', count: 4 }, { game: 't5', mapSlug: 'factory', count: 2 }
] });
const preview = previewCollection(blueprint.id);
expect(preview.entries.map(({ count, existing, missing }) => ({ count, existing, missing }))).toEqual([
{ count: 4, existing: 2, missing: 2 }, { count: 2, existing: 0, missing: 2 }
]);
expect(preview.servers.map(server => server.name)).toEqual(['t5_zombie_theater_3', 't5_zombie_theater_4', 't5_zombie_factory_1', 't5_zombie_factory_2']);
const queued = queueCollection({ previewId: preview.id, keys: keysFor(preview) });
expect(getCollection(queued.collectionId!)?.collection.total).toBe(4);
expect(previewCollection(blueprint.id).servers).toEqual([]);
});
it('keeps excess and unrelated servers, including stopped servers, without requiring launch settings for a no-op', async () => {
const { createServer } = await import('../../src/lib/server/fleet/create.js');
for (let i = 0; i < 2; i++) await createServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'existing-key' });
const { db } = await import('../../src/lib/server/db/index.js');
db().prepare("UPDATE servers SET container_state = 'stopped'").run();
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
const { config } = await import('../../src/lib/server/config.js');
// Fixture-only configuration changes simulate missing integrations after creation.
Object.assign(config.rcon, { password: '' });
Object.assign(config.komodo, { targetServer: '' });
const blueprint = saveBlueprint({ name: 'One Kino', entries: [{ game: 't5', mapSlug: 'theater', count: 1 }] });
const preview = previewCollection(blueprint.id);
expect(preview.entries[0]).toMatchObject({ existing: 2, missing: 0 });
expect(preview).toMatchObject({ ready: true, blockers: [], servers: [] });
expect(queueCollection({ previewId: preview.id })).toEqual({ ok: true, collectionId: null });
expect(db().prepare('SELECT * FROM server_launch_collections').all()).toEqual([]);
expect(db().prepare('SELECT * FROM servers').all()).toHaveLength(2);
});
it('rechecks shrinking deficits at submission and rejects an older additive preview', async () => {
const { preview, blueprint } = await previewLineup();
const { queueCreateServer } = await import('../../src/lib/server/fleet/create.js');
const { queueCollection, previewCollection } = await import('../../src/lib/server/fleet/collections.js');
const { db } = await import('../../src/lib/server/db/index.js');
queueCreateServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'existing-key' });
expect(() => queueCollection({ previewId: preview.id, keys: keysFor(preview) })).toThrow(/changed/);
expect(db().prepare('SELECT * FROM server_creation_jobs').all()).toHaveLength(1);
const fresh = previewCollection(blueprint.id);
expect(fresh.servers).toHaveLength(4);
const extra = [...fresh.servers, { ...fresh.servers[0], name: 't5_zombie_theater_20', port: 28980 }];
db().prepare('UPDATE server_collection_previews SET servers = ? WHERE id = ?').run(JSON.stringify(extra), fresh.id);
expect(() => queueCollection({ previewId: fresh.id, keys: keysFor(fresh) })).toThrow(/changed/);
});
it('matches exact T6 variants and excludes failed creations and pending destructions', async () => {
const { createServer, queueCreateServer, destroyServer } = await import('../../src/lib/server/fleet/create.js');
const { finishDeployment } = await import('../../src/lib/server/fleet/deployments.js');
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
const { previewCollection } = await import('../../src/lib/server/fleet/collections.js');
const farm = await createServer({ game: 't6', gameMode: 'zombie', mapSlug: 'farm', plutoKey: 'farm-key' });
const town = await createServer({ game: 't6', gameMode: 'zombie', mapSlug: 'town', plutoKey: 'town-key' });
if (!farm.ok || !town.ok) throw new Error('Fixture creation failed');
destroyServer(town.name);
const failed = queueCreateServer({ game: 't6', gameMode: 'zombie', mapSlug: 'town', plutoKey: 'failed-key' });
if (!failed.ok) throw new Error(failed.message);
finishDeployment(failed.deploymentId, { ok: false, message: 'Fixture failure' });
const blueprint = saveBlueprint({ name: 'Town target', entries: [{ game: 't6', mapSlug: 'town', count: 1 }] });
const preview = previewCollection(blueprint.id);
expect(preview.entries[0]).toMatchObject({ existing: 0, missing: 1 });
expect(preview.servers[0].name).toBe('t6_zombie_town_2');
});
it('queues all five servers, persists one group, and completes each independently', async () => {
const { preview } = await previewLineup();
const { queueCollection, getCollection, listCollections } = await import('../../src/lib/server/fleet/collections.js');
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
const { getDeployment } = await import('../../src/lib/server/fleet/deployments.js');
const { closeDb } = await import('../../src/lib/server/db/index.js');
const { komodoCreateDeployment } = await import('../../src/lib/server/integrations/komodo.js');
const queued = queueCollection({ previewId: preview.id, keys: keysFor(preview) });
expect(komodoCreateDeployment).not.toHaveBeenCalled();
expect(getCollection(queued.collectionId!)?.collection).toMatchObject({ total: 5, active: 5, done: 0, failed: 0, queued: 5, provisioning: 0, verifying: 0 });
expect(getCollection(queued.collectionId!)?.servers.map(server => server.queuePosition)).toEqual([1, 2, 3, 4, 5]);
closeDb();
await processCreations();
expect(getCollection(queued.collectionId!)?.collection).toMatchObject({ queued: 4, provisioning: 0, verifying: 1, done: 0 });
expect(getCollection(queued.collectionId!)?.queue).toMatchObject({ concurrency: 1, queued: 4, active: [{ server: preview.servers[0].name, state: 'awaiting-health' }] });
expect(getCollection(queued.collectionId!)?.servers.map(server => server.queuePosition)).toEqual([null, 1, 2, 3, 4]);
for (let pass = 0; pass < preview.servers.length; pass++) await processCreations();
const detail = getCollection(queued.collectionId!)!;
expect(detail.collection).toMatchObject({ total: 5, active: 0, done: 5, failed: 0 });
expect(listCollections()).toHaveLength(1);
expect(komodoCreateDeployment).toHaveBeenCalledTimes(5);
expect(getDeployment(detail.servers[0].id)?.collection).toEqual({ id: queued.collectionId, name: 'Friday Zombies' });
expect(await readFile(join(fixture.t6, 't6_zombie_town_2', 'configs/electric_cherry_map.cfg'), 'utf8')).toContain('execgts zm_standard_town.cfg map zm_transit');
expect(JSON.stringify(detail)).not.toContain('key-t5');
});
it('rolls back all earlier jobs and key changes if the final server has no valid key', async () => { it('rolls back all earlier jobs and key changes if the final server has no valid key', async () => {
const { preview } = await previewLineup(); const { preview } = await previewLineup();
@@ -264,7 +98,7 @@ describe('atomic collection launches', () => {
saveServerKey(preview.servers[0].name!, 'original-key'); saveServerKey(preview.servers[0].name!, 'original-key');
const keys = keysFor(preview); const keys = keysFor(preview);
delete keys[preview.servers[4].name!]; delete keys[preview.servers[4].name!];
expect(() => queueCollection({ previewId: preview.id, keys })).toThrow(/No saved key/); expect(() => queueCollection({ previewId: preview.id, keys })).toThrow(/Save a Plutonium server key/);
for (const table of ['server_launch_collections', 'server_collection_members', 'server_creation_jobs', 'server_deployments', 'server_launch_previews', 'server_launch_recipes']) { for (const table of ['server_launch_collections', 'server_collection_members', 'server_creation_jobs', 'server_deployments', 'server_launch_previews', 'server_launch_recipes']) {
expect(db().prepare(`SELECT * FROM ${table}`).all()).toEqual([]); expect(db().prepare(`SELECT * FROM ${table}`).all()).toEqual([]);
} }
@@ -276,18 +110,6 @@ describe('atomic collection launches', () => {
expect(queueCollection({ previewId: preview.id, keys: keysFor(preview) }).ok).toBe(true); expect(queueCollection({ previewId: preview.id, keys: keysFor(preview) }).ok).toBe(true);
}); });
it('rejects changed capacity before saving keys or partially queueing the lineup', async () => {
const { preview } = await previewLineup();
const { queueCreateServer } = await import('../../src/lib/server/fleet/create.js');
const { queueCollection } = await import('../../src/lib/server/fleet/collections.js');
const { db } = await import('../../src/lib/server/db/index.js');
queueCreateServer({ gameMode: 'zombie', mapSlug: 'cosmodrome', plutoKey: 'other-server-key' });
expect(() => queueCollection({ previewId: preview.id, keys: keysFor(preview) })).toThrow(/availability changed/);
expect(db().prepare('SELECT * FROM server_creation_jobs').all()).toHaveLength(1);
expect(db().prepare('SELECT * FROM server_keys').all()).toHaveLength(1);
expect(db().prepare('SELECT * FROM server_launch_collections').all()).toEqual([]);
});
it('reuses saved keys and returns the same collection on resubmission after restart and expiry', async () => { it('reuses saved keys and returns the same collection on resubmission after restart and expiry', async () => {
const { preview, blueprint } = await previewLineup(); const { preview, blueprint } = await previewLineup();
const { queueCollection, previewCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js'); const { queueCollection, previewCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
@@ -309,52 +131,6 @@ describe('atomic collection launches', () => {
expect(db().prepare('SELECT * FROM server_launch_collections').all()).toHaveLength(1); expect(db().prepare('SELECT * FROM server_launch_collections').all()).toHaveLength(1);
}); });
it('keeps the reviewed lineup after blueprint edits and deletion while using current promotions', async () => {
const { createScript, uploadNewVersion } = await import('../../src/lib/server/fleet/scripts.js');
const { db } = await import('../../src/lib/server/db/index.js');
const { script, version } = createScript({ name: 'Classic perks', kind: 'script', scopeType: 'game', scopeValue: 't5', targetRelPath: 'perks.gsc', content: Buffer.from('original') });
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(version.id, script.id);
const { preview, blueprint } = await previewLineup();
const { updateBlueprint, deleteBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
const { queueCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
const newer = uploadNewVersion(script.id, Buffer.from('new version'));
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(newer.id, script.id);
updateBlueprint(blueprint.id, { name: 'Changed', entries: [{ ...entries[0], count: 1 }] });
deleteBlueprint(blueprint.id);
const queued = queueCollection({ previewId: preview.id, keys: keysFor(preview) });
const queuedDetail = getCollection(queued.collectionId!)!;
const { recipeForDeployment } = await import('../../src/lib/server/fleet/launch-recipes.js');
for (const server of queuedDetail.servers.filter(server => server.server.startsWith('t5_'))) {
expect(recipeForDeployment(server.id)?.recipe.files).toContainEqual(expect.objectContaining({ path: 'perks.gsc', versionId: newer.id }));
}
const newest = uploadNewVersion(script.id, Buffer.from('promoted while queued'));
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(newest.id, script.id);
uploadNewVersion(script.id, Buffer.from('unpromoted draft'));
for (let pass = 0; pass <= preview.servers.length; pass++) await processCreations();
expect(getCollection(queued.collectionId!)?.collection).toMatchObject({ name: 'Friday Zombies', total: 5, done: 5 });
for (const name of ['t5_zombie_theater_1', 't5_zombie_theater_2', 't5_zombie_moon_1']) {
expect(await readFile(join(fixture.t5, name, 'runtime/plutonium/storage/t5/raw/scripts/sp/perks.gsc'), 'utf8')).toBe('promoted while queued');
}
});
it('keeps a provisioning failure separate while the other servers complete', async () => {
const { preview } = await previewLineup();
const { queueCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
const { processCreations } = await import('../../src/lib/server/fleet/create.js');
const { komodoCreateDeployment } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoCreateDeployment).mockResolvedValueOnce({ ok: false, kind: 'http-error', message: 'Host rejected this container' });
const queued = queueCollection({ previewId: preview.id, keys: keysFor(preview) });
await processCreations();
expect(getCollection(queued.collectionId!)?.collection).toMatchObject({ queued: 4, failed: 1, done: 0 });
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
for (let pass = 0; pass < preview.servers.length; pass++) await processCreations();
const detail = getCollection(queued.collectionId!)!;
expect(detail.collection).toMatchObject({ total: 5, active: 0, done: 4, failed: 1 });
expect(detail.servers[0]).toMatchObject({ state: 'failed', error: expect.stringContaining('Host rejected') });
expect(detail.servers.slice(1).every(server => server.state === 'done')).toBe(true);
});
it('validates collection API payloads and rejects an unused expired preview', async () => { it('validates collection API payloads and rejects an unused expired preview', async () => {
const { preview, blueprint } = await previewLineup(); const { preview, blueprint } = await previewLineup();
const { POST: singlePreview } = await import('../../src/routes/api/deployments/preview/+server.js'); const { POST: singlePreview } = await import('../../src/routes/api/deployments/preview/+server.js');
@@ -382,52 +158,6 @@ describe('matching a blueprint exactly', () => {
return saveBlueprint({ name: 'Exact lineup', entries }); return saveBlueprint({ name: 'Exact lineup', entries });
} }
it('keeps the lowest deployed lobbies, removes excess and unrelated maps, and adds only the shortfall', async () => {
const { createServer, processCreations, processDestructions } = await import('../../src/lib/server/fleet/create.js');
for (const mapSlug of ['theater', 'theater', 'moon']) await createServer({ gameMode: 'zombie', mapSlug, plutoKey: 'existing' });
const { db } = await import('../../src/lib/server/db/index.js');
const before = db().prepare("SELECT * FROM servers WHERE name = 't5_zombie_theater_1'").get();
const { previewCollection, queueCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
const plan = await blueprint([{ game: 't5', mapSlug: 'theater', count: 1 }, { game: 't6', mapSlug: 'town', count: 1 }]);
const preview = previewCollection(plan.id, 'exact');
expect(preview.retainedServers).toEqual(['t5_zombie_theater_1']);
expect(preview.removals.map(server => server.name)).toEqual(['t5_zombie_moon_1', 't5_zombie_theater_2']);
expect(preview.servers.map(server => server.name)).toEqual(['t6_zombie_town_1']);
const { komodoDestroyDeployment, komodoCreateDeployment, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoCreateDeployment).mockClear();
const result = queueCollection({ previewId: preview.id, keys: keysFor(preview) });
expect(komodoDestroyDeployment).not.toHaveBeenCalled();
expect(komodoCreateDeployment).not.toHaveBeenCalled();
expect(getCollection(result.collectionId!)?.collection).toMatchObject({ mode: 'exact', total: 3, active: 3, queued: 3 });
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: false, reason: 'gone' }).mockResolvedValueOnce({ ok: false, reason: 'gone' });
await processDestructions();
await processCreations();
await processCreations();
expect(getCollection(result.collectionId!)?.collection).toMatchObject({ total: 3, done: 3, active: 0 });
expect(getCollection(result.collectionId!)?.servers.filter(server => server.action === 'destroy')).toHaveLength(2);
expect(db().prepare("SELECT * FROM servers WHERE name = 't5_zombie_theater_1'").get()).toEqual(before);
expect(db().prepare('SELECT name FROM servers ORDER BY name').all()).toEqual([{ name: 't5_zombie_theater_1' }, { name: 't6_zombie_town_1' }]);
expect(komodoDestroyDeployment).not.toHaveBeenCalledWith('t5_zombie_theater_1');
});
it('supports removal-only plans, shares existing removals, and records retained servers', async () => {
const { createServer } = await import('../../src/lib/server/fleet/create.js');
for (let i = 0; i < 2; i++) await createServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'existing' });
const plan = await blueprint([{ game: 't5', mapSlug: 'theater', count: 1 }]);
const { previewCollection, queueCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
const first = previewCollection(plan.id, 'exact');
expect(first.servers).toEqual([]);
const a = queueCollection({ previewId: first.id });
const b = queueCollection({ previewId: previewCollection(plan.id, 'exact').id });
const firstDetail = getCollection(a.collectionId!)!;
expect(firstDetail.retainedServers).toEqual(['t5_zombie_theater_1']);
expect(firstDetail.servers).toHaveLength(1);
expect(getCollection(b.collectionId!)?.servers[0].id).toBe(firstDetail.servers[0].id);
expect(firstDetail.servers[0]).toMatchObject({ action: 'destroy', state: 'waiting', queuePosition: null });
const { getDeployment } = await import('../../src/lib/server/fleet/deployments.js');
expect(getDeployment(firstDetail.servers[0].id)?.collection).toBeTruthy();
});
it('cancels unstarted excess creations without booting them and retains their saved keys', async () => { it('cancels unstarted excess creations without booting them and retains their saved keys', async () => {
const { queueCreateServer } = await import('../../src/lib/server/fleet/create.js'); const { queueCreateServer } = await import('../../src/lib/server/fleet/create.js');
const first = queueCreateServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'first' }); const first = queueCreateServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'first' });
@@ -442,21 +172,6 @@ describe('matching a blueprint exactly', () => {
expect(getCollection(queued.collectionId!)?.collection).toMatchObject({ done: 1, total: 1, active: 0 }); expect(getCollection(queued.collectionId!)?.collection).toMatchObject({ done: 1, total: 1, active: 0 });
const { getServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); const { getServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
expect(getServerKey(second.name)).toBe('second'); expect(getServerKey(second.name)).toBe('second');
const { komodoCreateDeployment, komodoDestroyDeployment, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
expect(komodoCreateDeployment).not.toHaveBeenCalled();
expect(komodoDestroyDeployment).not.toHaveBeenCalled();
});
it('rejects fleet changes even when the create shortfall stays the same', async () => {
const { createServer, queueCreateServer } = await import('../../src/lib/server/fleet/create.js');
await createServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'existing' });
const plan = await blueprint([{ game: 't5', mapSlug: 'theater', count: 1 }]);
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
const preview = previewCollection(plan.id, 'exact');
queueCreateServer({ gameMode: 'zombie', mapSlug: 'moon', plutoKey: 'new' });
expect(() => queueCollection({ previewId: preview.id })).toThrow(/fleet changed/);
const { db } = await import('../../src/lib/server/db/index.js');
expect(db().prepare("SELECT id FROM server_deployments WHERE action = 'destroy'").all()).toEqual([]);
}); });
it('rolls back removals and creation cancellations when a required key is missing', async () => { it('rolls back removals and creation cancellations when a required key is missing', async () => {
@@ -466,128 +181,13 @@ describe('matching a blueprint exactly', () => {
const plan = await blueprint([{ game: 't5', mapSlug: 'theater', count: 1 }]); const plan = await blueprint([{ game: 't5', mapSlug: 'theater', count: 1 }]);
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js'); const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
const preview = previewCollection(plan.id, 'exact'); const preview = previewCollection(plan.id, 'exact');
expect(() => queueCollection({ previewId: preview.id })).toThrow(/No saved key/); expect(() => queueCollection({ previewId: preview.id })).toThrow(/Save a Plutonium server key/);
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
expect(db().prepare('SELECT state FROM server_deployments WHERE id = ?').get(old.deploymentId)).toEqual({ state: 'queued' }); expect(db().prepare('SELECT state FROM server_deployments WHERE id = ?').get(old.deploymentId)).toEqual({ state: 'queued' });
for (const table of ['server_collection_removals', 'server_launch_collections', 'server_creation_dependencies']) expect(db().prepare(`SELECT * FROM ${table}`).all()).toEqual([]); for (const table of ['server_collection_removals', 'server_launch_collections', 'server_creation_dependencies']) expect(db().prepare(`SELECT * FROM ${table}`).all()).toEqual([]);
expect(db().prepare("SELECT id FROM server_deployments WHERE action = 'destroy'").all()).toEqual([]); expect(db().prepare("SELECT id FROM server_deployments WHERE action = 'destroy'").all()).toEqual([]);
}); });
it('reuses a full fleet port only after empty-server removal, across restart and repeated submission', async () => {
vi.stubEnv('SERVER_PORT_MAX', '28960');
const { createServer, processCreations, processDestructions } = await import('../../src/lib/server/fleet/create.js');
await createServer({ gameMode: 'zombie', mapSlug: 'moon', plutoKey: 'existing' });
await processCreations();
const plan = await blueprint([{ game: 't5', mapSlug: 'theater', count: 1 }]);
const { previewCollection, queueCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
const preview = previewCollection(plan.id, 'exact');
expect(preview).toMatchObject({ ready: true, servers: [{ port: 28960 }] });
const result = queueCollection({ previewId: preview.id, keys: keysFor(preview) });
const { closeDb } = await import('../../src/lib/server/db/index.js');
closeDb();
expect(queueCollection({ previewId: preview.id })).toEqual(result);
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
const { komodoCreateDeployment, komodoDestroyDeployment, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoCreateDeployment).mockClear();
for (const players of [2, null]) {
vi.mocked(rconGetPlayerCount).mockResolvedValueOnce(players);
await processDestructions();
await processCreations();
expect(komodoDestroyDeployment).not.toHaveBeenCalled();
expect(komodoCreateDeployment).not.toHaveBeenCalled();
expect(getCollection(result.collectionId!)?.servers.find(server => server.action === 'create')?.stage).toContain('Waiting for removal');
}
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: false, reason: 'gone' });
await processDestructions();
await processCreations();
await processCreations();
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
expect(getCollection(result.collectionId!)?.collection).toMatchObject({ total: 2, done: 2, active: 0 });
});
it('does not reuse a port after a removal is cancelled', async () => {
vi.stubEnv('SERVER_PORT_MAX', '28960');
const { createServer, processCreations } = await import('../../src/lib/server/fleet/create.js');
await createServer({ gameMode: 'zombie', mapSlug: 'moon', plutoKey: 'existing' });
await processCreations();
const plan = await blueprint([{ game: 't5', mapSlug: 'theater', count: 1 }]);
const { previewCollection, queueCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
const preview = previewCollection(plan.id, 'exact');
const result = queueCollection({ previewId: preview.id, keys: keysFor(preview) });
const { cancelRemoval } = await import('../../src/lib/server/fleet/deployments.js');
expect(cancelRemoval(getCollection(result.collectionId!)!.servers.find(server => server.action === 'destroy')!.id)).toBe(true);
const { komodoCreateDeployment } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoCreateDeployment).mockClear();
await processCreations();
expect(komodoCreateDeployment).not.toHaveBeenCalled();
expect(getCollection(result.collectionId!)?.collection.failed).toBe(1);
expect(getCollection(result.collectionId!)?.servers.find(server => server.action === 'create')?.stage).toContain('(cancelled)');
});
it('allows independent launches to pass a failed removal without exceeding the startup limit', async () => {
const { createServer, processCreations, processDestructions } = await import('../../src/lib/server/fleet/create.js');
await createServer({ gameMode: 'zombie', mapSlug: 'moon', plutoKey: 'existing' });
await processCreations();
const plan = await blueprint([{ game: 't5', mapSlug: 'theater', count: 2 }]);
const { previewCollection, queueCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
const preview = previewCollection(plan.id, 'exact');
const result = queueCollection({ previewId: preview.id, keys: keysFor(preview) });
const { komodoCreateDeployment, komodoDestroyDeployment } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoCreateDeployment).mockClear();
vi.mocked(komodoDestroyDeployment).mockResolvedValueOnce({ ok: false, kind: 'http-error', message: 'Could not stop container' });
await processDestructions();
await processCreations();
const creates = getCollection(result.collectionId!)!.servers.filter(server => server.action === 'create');
expect(creates[0]).toMatchObject({ state: 'queued', stage: expect.stringContaining('(failed)') });
expect(creates[1]).toMatchObject({ state: 'awaiting-health' });
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
expect(getCollection(result.collectionId!)!.collection).toMatchObject({ failed: 1, queued: 1, verifying: 1 });
});
it('rejects insufficient final capacity before cancelling any existing creation', async () => {
vi.stubEnv('SERVER_PORT_MAX', '28960');
const { queueCreateServer } = await import('../../src/lib/server/fleet/create.js');
const old = queueCreateServer({ gameMode: 'zombie', mapSlug: 'moon', plutoKey: 'existing' });
if (!old.ok) throw new Error(old.message);
const plan = await blueprint([{ game: 't5', mapSlug: 'theater', count: 2 }]);
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
const preview = previewCollection(plan.id, 'exact');
expect(preview.ready).toBe(false);
expect(() => queueCollection({ previewId: preview.id, keys: keysFor(preview) })).toThrow(/not enough free game ports/);
const { db } = await import('../../src/lib/server/db/index.js');
expect(db().prepare('SELECT state FROM server_deployments WHERE id = ?').get(old.deploymentId)).toEqual({ state: 'queued' });
expect(db().prepare('SELECT * FROM server_launch_collections').all()).toEqual([]);
});
it('requires a new review when a pending removal is cancelled', async () => {
const { createServer, destroyServer } = await import('../../src/lib/server/fleet/create.js');
const old = await createServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'existing' });
if (!old.ok) throw new Error(old.message);
const removal = destroyServer(old.name);
if (!removal.ok) throw new Error(removal.message);
const plan = await blueprint([{ game: 't5', mapSlug: 'theater', count: 1 }]);
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
const preview = previewCollection(plan.id, 'exact');
const { cancelRemoval } = await import('../../src/lib/server/fleet/deployments.js');
cancelRemoval(removal.deploymentId!);
expect(() => queueCollection({ previewId: preview.id, keys: keysFor(preview) })).toThrow(/fleet changed/);
expect(previewCollection(plan.id, 'exact')).toMatchObject({ servers: [], removals: [], retainedServers: [old.name] });
});
it('keeps T6 variants distinct and leaves a matching fleet untouched', async () => {
const { createServer } = await import('../../src/lib/server/fleet/create.js');
await createServer({ game: 't6', gameMode: 'zombie', mapSlug: 'town', plutoKey: 'town' });
const plan = await blueprint([{ game: 't6', mapSlug: 'town', count: 1 }]);
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
const noOp = previewCollection(plan.id, 'exact');
expect(noOp).toMatchObject({ retainedServers: ['t6_zombie_town_1'], removals: [], servers: [] });
expect(queueCollection({ previewId: noOp.id })).toEqual({ ok: true, collectionId: null });
await createServer({ game: 't6', gameMode: 'zombie', mapSlug: 'farm', plutoKey: 'farm' });
const preview = previewCollection(plan.id, 'exact');
expect(preview.retainedServers).toEqual(['t6_zombie_town_1']);
expect(preview.removals.map(server => server.name)).toEqual(['t6_zombie_farm_1']);
});
it('defaults old previews and collections to fill and validates the preview mode API', async () => { it('defaults old previews and collections to fill and validates the preview mode API', async () => {
const { blueprint: plan } = await previewLineup(); const { blueprint: plan } = await previewLineup();
const { db, closeDb } = await import('../../src/lib/server/db/index.js'); const { db, closeDb } = await import('../../src/lib/server/db/index.js');
-215
View File
@@ -1,215 +0,0 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { join } from 'node:path';
import { newFixture, type Fixture } from '../helpers/fixture.js';
vi.mock('../../src/lib/server/integrations/komodo.js', () => ({
komodoCreateDeployment: vi.fn(),
komodoDeploy: vi.fn(),
komodoDestroyDeployment: vi.fn(),
komodoDeleteDeployment: vi.fn(),
komodoInspectHealthy: vi.fn(),
komodoEnsureTag: vi.fn(),
komodoUpdateTagsOnResource: vi.fn()
}));
vi.mock('../../src/lib/server/integrations/rcon.js', () => ({
rconGetPlayerCount: vi.fn(), rconStatusResult: vi.fn()
}));
vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}) }));
function deferred() {
let resolve!: () => void;
let reject!: (error: Error) => void;
const promise = new Promise<void>((done, fail) => { resolve = done; reject = fail; });
return { promise, resolve, reject };
}
function command() {
return { started: deferred(), response: deferred() };
}
const servers = Array.from({ length: 4 }, (_, index) => `t5_zombie_theater_${index + 1}`);
let fixture: Fixture;
let commands: Map<string, { destroy: ReturnType<typeof command>; delete: ReturnType<typeof command> }>;
let passes: Promise<void>[];
let runningAtSubmission: number[];
beforeEach(async () => {
vi.resetModules(); vi.resetAllMocks();
fixture = await newFixture(servers);
for (const [key, value] of Object.entries({ KOMODO_ADDRESS: 'http://komodo.test', KOMODO_API_KEY: 'test',
KOMODO_API_SECRET: 'test', KOMODO_TARGET_SERVER: 'test-host', RCON_PASSWORD: 'test' })) vi.stubEnv(key, value);
commands = new Map(servers.map(server => [server, { destroy: command(), delete: command() }]));
passes = [];
runningAtSubmission = [];
const { db } = await import('../../src/lib/server/db/index.js');
for (const [index, server] of servers.entries()) {
db().prepare(`INSERT INTO servers (name, komodo_type, komodo_name, game, map, game_mode, map_slug, game_port, host_dir)
VALUES (?, 'deployment', ?, 't5', 'zombie_theater', 'zombie', 'theater', ?, ?)`)
.run(server, server, 28960 + index, join(fixture.t5, server));
}
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
const { komodoDestroyDeployment, komodoDeleteDeployment, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(rconGetPlayerCount).mockResolvedValue(0);
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'exited', reason: 'stopped' });
vi.mocked(komodoDestroyDeployment).mockImplementation(async server => {
runningAtSubmission.push((db().prepare("SELECT COUNT(*) AS n FROM server_deployments WHERE action = 'destroy' AND state = 'running'")
.get() as { n: number }).n);
const gate = commands.get(server)!.destroy;
gate.started.resolve();
await gate.response.promise;
return { ok: true };
});
vi.mocked(komodoDeleteDeployment).mockImplementation(async server => {
const gate = commands.get(server)!.delete;
gate.started.resolve();
await gate.response.promise;
return { ok: true };
});
});
afterEach(async () => {
for (const server of servers) complete(server);
await Promise.allSettled(passes);
await fixture.cleanup();
vi.unstubAllEnvs();
vi.restoreAllMocks();
});
function complete(server: string) {
commands.get(server)!.destroy.response.resolve();
commands.get(server)!.delete.response.resolve();
}
function track(pass: Promise<void>) {
passes.push(pass);
return pass;
}
async function queue(count = servers.length) {
const { destroyServer } = await import('../../src/lib/server/fleet/create.js');
return servers.slice(0, count).map(server => {
const result = destroyServer(server);
if (!result.ok) throw new Error(result.message);
return result.deploymentId!;
});
}
async function states() {
const { db } = await import('../../src/lib/server/db/index.js');
return (db().prepare("SELECT state FROM server_deployments WHERE action = 'destroy' ORDER BY rowid")
.all() as { state: string }[]).map(row => row.state);
}
describe('destruction concurrency', () => {
it('defaults to one full removal at a time and refills the slot in the same pass', async () => {
await queue(2);
const { fleetSettings } = await import('../../src/lib/server/settings.js');
const { processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { komodoDestroyDeployment } = await import('../../src/lib/server/integrations/komodo.js');
expect(fleetSettings().destroyConcurrency).toBe(1);
const pass = track(processDestructions());
await commands.get(servers[0])!.destroy.started.promise;
expect(komodoDestroyDeployment).toHaveBeenCalledTimes(1);
expect(await states()).toEqual(['running', 'waiting']);
complete(servers[0]);
await commands.get(servers[1])!.destroy.started.promise;
expect(await states()).toEqual(['done', 'running']);
complete(servers[1]);
await pass;
expect(await states()).toEqual(['done', 'done']);
expect(runningAtSubmission).toEqual([1, 1]);
});
it('runs two removals concurrently, holds slots through deletion, and coalesces overlapping passes', async () => {
await queue();
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
saveFleetSettings({ ...fleetSettings(), destroyConcurrency: 2 });
const { processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { komodoDestroyDeployment, komodoDeleteDeployment } = await import('../../src/lib/server/integrations/komodo.js');
const pass = track(processDestructions());
await Promise.all(servers.slice(0, 2).map(server => commands.get(server)!.destroy.started.promise));
const overlapping = track(processDestructions());
expect(komodoDestroyDeployment).toHaveBeenCalledTimes(2);
expect(await states()).toEqual(['running', 'running', 'waiting', 'waiting']);
commands.get(servers[0])!.destroy.response.resolve();
await commands.get(servers[0])!.delete.started.promise;
expect(komodoDestroyDeployment).toHaveBeenCalledTimes(2);
commands.get(servers[0])!.delete.response.resolve();
await commands.get(servers[2])!.destroy.started.promise;
expect(await states()).toEqual(['done', 'running', 'running', 'waiting']);
complete(servers[1]);
await commands.get(servers[3])!.destroy.started.promise;
complete(servers[2]); complete(servers[3]);
await Promise.all([pass, overlapping]);
expect(await states()).toEqual(['done', 'done', 'done', 'done']);
expect(vi.mocked(komodoDestroyDeployment).mock.calls.map(([server]) => server)).toEqual(servers);
expect(komodoDeleteDeployment).toHaveBeenCalledTimes(4);
expect(Math.max(...runningAtSubmission)).toBe(2);
});
it('lets active removals finish after lowering the limit and applies it to new admissions', async () => {
await queue();
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
saveFleetSettings({ ...fleetSettings(), destroyConcurrency: 2 });
const { processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { komodoDestroyDeployment } = await import('../../src/lib/server/integrations/komodo.js');
const pass = track(processDestructions());
await Promise.all(servers.slice(0, 2).map(server => commands.get(server)!.destroy.started.promise));
saveFleetSettings({ ...fleetSettings(), destroyConcurrency: 1 });
complete(servers[0]);
await vi.waitFor(async () => expect(await states()).toEqual(['done', 'running', 'waiting', 'waiting']));
expect(komodoDestroyDeployment).toHaveBeenCalledTimes(2);
complete(servers[1]);
await commands.get(servers[2])!.destroy.started.promise;
expect(komodoDestroyDeployment).toHaveBeenCalledTimes(3);
expect(await states()).toEqual(['done', 'done', 'running', 'waiting']);
complete(servers[2]);
await commands.get(servers[3])!.destroy.started.promise;
complete(servers[3]);
await pass;
expect(runningAtSubmission.slice(2)).toEqual([1, 1]);
});
it.each(['rejected', 'thrown'] as const)('releases capacity after a %s teardown failure', async failure => {
await queue(3);
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
saveFleetSettings({ ...fleetSettings(), destroyConcurrency: 2 });
const { processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { komodoDestroyDeployment, komodoDeleteDeployment } = await import('../../src/lib/server/integrations/komodo.js');
if (failure === 'rejected') {
vi.mocked(komodoDestroyDeployment).mockImplementationOnce(async () => {
commands.get(servers[0])!.destroy.started.resolve();
await commands.get(servers[0])!.destroy.response.promise;
return { ok: false, kind: 'http-error', message: 'Teardown rejected' };
});
}
const pass = track(processDestructions());
await Promise.all(servers.slice(0, 2).map(server => commands.get(server)!.destroy.started.promise));
if (failure === 'thrown') commands.get(servers[0])!.destroy.response.reject(new Error('Connection lost'));
else commands.get(servers[0])!.destroy.response.resolve();
await commands.get(servers[2])!.destroy.started.promise;
expect(await states()).toEqual(['failed', 'running', 'running']);
complete(servers[1]); complete(servers[2]);
await pass;
expect(await states()).toEqual(['failed', 'done', 'done']);
expect(komodoDestroyDeployment).toHaveBeenCalledTimes(3);
expect(komodoDeleteDeployment).not.toHaveBeenCalledWith(servers[0]);
});
it('skips occupied and unknown servers while allowing later empty and stopped servers', async () => {
await queue();
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
saveFleetSettings({ ...fleetSettings(), destroyConcurrency: 2 });
const { processDestructions } = await import('../../src/lib/server/fleet/create.js');
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
const { komodoDestroyDeployment, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(rconGetPlayerCount).mockImplementation(async server => server === servers[0] ? 2 : server === servers[2] ? 0 : null);
vi.mocked(komodoInspectHealthy).mockImplementation(async server => server === servers[1]
? { ok: false, reason: 'Host unavailable' } : { ok: false, state: 'exited', reason: 'stopped' });
for (const server of servers) complete(server);
await track(processDestructions());
expect(await states()).toEqual(['waiting', 'waiting', 'done', 'done']);
expect(vi.mocked(komodoDestroyDeployment).mock.calls.map(([server]) => server)).toEqual(servers.slice(2));
expect(rconGetPlayerCount).toHaveBeenCalledTimes(4);
});
});
+1 -1
View File
@@ -31,7 +31,7 @@ async function applyDriftDeployment(input: import('../../src/lib/server/fleet/dr
beforeEach(async () => { beforeEach(async () => {
vi.resetModules(); vi.resetModules();
fixture = await newFixture(['t5_zombie_theater_1']); fixture = await newFixture(['t5_zombie_theater_1']);
delete process.env.KOMODO_ADDRESS;
delete process.env.RCON_PASSWORD; delete process.env.RCON_PASSWORD;
// Ensure the three per-kind destinations exist under the fixture tree. // Ensure the three per-kind destinations exist under the fixture tree.
await mkdir(join(fixture.t5, 't5_zombie_theater_1', 'runtime/plutonium/storage/t5/raw/scripts/sp'), { recursive: true }); await mkdir(join(fixture.t5, 't5_zombie_theater_1', 'runtime/plutonium/storage/t5/raw/scripts/sp'), { recursive: true });
+1 -1
View File
@@ -37,7 +37,7 @@ async function setup() {
describe('fleet event refreshes', () => { describe('fleet event refreshes', () => {
it('refreshes server history through one scoped request without replacing the fleet snapshot', async () => { it('refreshes server history through one scoped request without replacing the fleet snapshot', async () => {
const store = await import('../../src/lib/ui/stores/fleet.js'); const store = await import('../../src/lib/ui/stores/fleet.js');
const integrations = { rcon: true, komodo: true, rconTimeoutMs: 2000, defaultRconHost: '' }; const integrations = { rcon: true, rconTimeoutMs: 2000, defaultRconHost: '' };
const snapshot = { servers: [], operations: [], integrations, serverNow: Date.now() }; const snapshot = { servers: [], operations: [], integrations, serverNow: Date.now() };
store.hydrateFleet(snapshot); store.hydrateFleet(snapshot);
const originalFleet = get(store.fleet); const originalFleet = get(store.fleet);
+2 -3
View File
@@ -54,7 +54,6 @@ describe('Fleet discovery and match freshness', () => {
}); });
}); });
describe('persisted Fleet operations', () => { describe('persisted Fleet operations', () => {
let fixture: Fixture; let fixture: Fixture;
beforeEach(async () => { vi.resetModules(); fixture = await newFixture(); }); beforeEach(async () => { vi.resetModules(); fixture = await newFixture(); });
@@ -75,7 +74,7 @@ describe('persisted Fleet operations', () => {
it('recovers waiting and recovering campaigns without needing a live event', async () => { it('recovers waiting and recovering campaigns without needing a live event', async () => {
const { db, closeDb } = await import('../../src/lib/server/db/index.js'); const { db, closeDb } = await import('../../src/lib/server/db/index.js');
const handle = db(); const handle = db();
handle.prepare("INSERT INTO servers (name, komodo_name) VALUES ('server', 'server')").run(); handle.prepare("INSERT INTO servers (name) VALUES ('server')").run();
handle.prepare("INSERT INTO deployments (id, action, started_at) VALUES ('one', 'promote', ?)").run(iso(-2 * 86_400_000)); handle.prepare("INSERT INTO deployments (id, action, started_at) VALUES ('one', 'promote', ?)").run(iso(-2 * 86_400_000));
handle.prepare("INSERT INTO deployment_servers (deployment_id, server, files_changed, restart_state, players) VALUES ('one', 'server', 1, 'pending', 4)").run(); handle.prepare("INSERT INTO deployment_servers (deployment_id, server, files_changed, restart_state, players) VALUES ('one', 'server', 1, 'pending', 4)").run();
const { fleetOperations, serverOperations } = await import('../../src/lib/server/fleet/overview.js'); const { fleetOperations, serverOperations } = await import('../../src/lib/server/fleet/overview.js');
@@ -109,7 +108,7 @@ describe('persisted Fleet operations', () => {
it('surfaces orphaned recovery errors and beta gates', async () => { it('surfaces orphaned recovery errors and beta gates', async () => {
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const handle = db(); const handle = db();
handle.prepare("INSERT INTO servers (name, komodo_name) VALUES ('server', 'server')").run(); handle.prepare("INSERT INTO servers (name) VALUES ('server')").run();
handle.prepare("INSERT INTO restart_rollout (server, requested_at, error) VALUES ('server', ?, 'RCON not answering')").run(iso()); handle.prepare("INSERT INTO restart_rollout (server, requested_at, error) VALUES ('server', ?, 'RCON not answering')").run(iso());
const { fleetOperations, serverOperations } = await import('../../src/lib/server/fleet/overview.js'); const { fleetOperations, serverOperations } = await import('../../src/lib/server/fleet/overview.js');
expect(fleetOperations(now)[0]).toMatchObject({ kind: 'recovery', state: 'attention', detail: 'RCON not answering' }); expect(fleetOperations(now)[0]).toMatchObject({ kind: 'recovery', state: 'attention', detail: 'RCON not answering' });
+2 -2
View File
@@ -10,11 +10,11 @@ beforeEach(async () => {
fixture = await newFixture([]); fixture = await newFixture([]);
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
db().transaction(() => { db().transaction(() => {
const server = db().prepare('INSERT INTO servers(name, komodo_name) VALUES (?, ?)'); const server = db().prepare('INSERT INTO servers(name) VALUES (?)');
const operation = db().prepare(`INSERT INTO server_deployments(id,server,action,state,stage,started_at,finished_at,triggered_by) const operation = db().prepare(`INSERT INTO server_deployments(id,server,action,state,stage,started_at,finished_at,triggered_by)
VALUES(?,?,'restart',?,'Complete',?,?,'test')`); VALUES(?,?,'restart',?,'Complete',?,?,'test')`);
for (const name of ['one', 'two']) { for (const name of ['one', 'two']) {
server.run(name, name); server.run(name);
for (let i = 0; i < 100; i++) operation.run(`${name}-${i}`, name, 'done', iso(-i), iso(-i)); for (let i = 0; i < 100; i++) operation.run(`${name}-${i}`, name, 'done', iso(-i), iso(-i));
operation.run(`${name}-old`, name, 'done', iso(-86_400_001), iso(-86_400_001)); operation.run(`${name}-old`, name, 'done', iso(-86_400_001), iso(-86_400_001));
operation.run(`${name}-failed`, name, 'failed', iso(), iso()); operation.run(`${name}-failed`, name, 'failed', iso(), iso());
+1 -1
View File
@@ -16,7 +16,7 @@ let fetchMock: ReturnType<typeof vi.fn>;
beforeEach(async () => { beforeEach(async () => {
vi.resetModules(); vi.resetModules();
fixture = await newFixture(); fixture = await newFixture();
delete process.env.KOMODO_ADDRESS;
delete process.env.RCON_PASSWORD; delete process.env.RCON_PASSWORD;
fetchMock = vi.fn(async (input: string | URL) => String(input).startsWith('https://api.github.com/') fetchMock = vi.fn(async (input: string | URL) => String(input).startsWith('https://api.github.com/')
? Response.json(metadata()) : new Response(content)); ? Response.json(metadata()) : new Response(content));
+5 -36
View File
@@ -2,7 +2,6 @@ import { randomUUID } from 'node:crypto';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { newFixture, type Fixture } from '../helpers/fixture.js'; import { newFixture, type Fixture } from '../helpers/fixture.js';
vi.mock('../../src/lib/server/integrations/komodo.js', () => ({ komodoInspectHealthy: vi.fn() }));
vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconStatusResult: vi.fn() })); vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconStatusResult: vi.fn() }));
const server = 'managed'; const server = 'managed';
@@ -22,11 +21,12 @@ afterEach(async () => { vi.useRealTimers(); await fixture.cleanup(); vi.restoreA
async function setup() { async function setup() {
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const database = db(); const database = db();
database.prepare('INSERT INTO servers(name,komodo_name,created_at,game_port) VALUES(?,?,?,28960)') database.prepare('INSERT INTO servers(name,created_at,game_port) VALUES(?,?,28960)')
.run(server, server, new Date(now).toISOString()); .run(server, new Date(now).toISOString());
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js'); vi.spyOn(await import('../../src/lib/server/backends/native.js'), 'nativeHealth');
const { nativeHealth } = await import('../../src/lib/server/backends/native.js');
const { rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js'); const { rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js');
const inspect = vi.mocked(komodoInspectHealthy).mockResolvedValue(healthy); const inspect = vi.mocked(nativeHealth).mockResolvedValue(healthy);
const rcon = vi.mocked(rconStatusResult).mockResolvedValue(playing); const rcon = vi.mocked(rconStatusResult).mockResolvedValue(playing);
return { database, inspect, rcon, ...await import('../../src/lib/server/fleet/health-observations.js') }; return { database, inspect, rcon, ...await import('../../src/lib/server/fleet/health-observations.js') };
} }
@@ -228,37 +228,6 @@ describe('shared monitoring observations', () => {
expect(api.inspect).toHaveBeenCalledTimes(2); expect(api.inspect).toHaveBeenCalledTimes(2);
}); });
it('invalidates integration changes and accepts legacy port discovery by the successful RCON probe', async () => {
const api = await setup();
await api.inspectForMonitoring(server);
api.database.prepare("UPDATE servers SET komodo_name='replacement' WHERE name=?").run(server);
await api.inspectForMonitoring(server);
expect(api.inspect).toHaveBeenCalledTimes(2);
api.database.prepare('UPDATE servers SET game_port=NULL WHERE name=?').run(server);
api.rcon.mockImplementationOnce(async () => {
api.database.prepare('UPDATE servers SET game_port=28960 WHERE name=?').run(server);
return playing;
});
expect((await api.rconForMonitoring(server)).result).toEqual(playing);
await api.rconForMonitoring(server);
expect(api.rcon).toHaveBeenCalledTimes(1);
api.database.prepare('UPDATE servers SET game_port=28961 WHERE name=?').run(server);
await api.rconForMonitoring(server);
expect(api.rcon).toHaveBeenCalledTimes(2);
});
it.each(['inspect', 'rcon'] as const)('does not reuse %s after a newer manual observation', async kind => {
const api = await setup();
const call = kind === 'inspect' ? api.inspectForMonitoring : api.rconForMonitoring;
const request = kind === 'inspect' ? api.inspect : api.rcon;
const field = kind === 'inspect' ? 'container_checked_at' : 'last_checked_at';
await call(server);
vi.setSystemTime(now + 1000);
api.database.prepare(`UPDATE servers SET ${field}=? WHERE name=?`).run(new Date(now + 1000).toISOString(), server);
await call(server);
expect(request).toHaveBeenCalledTimes(2);
});
it('rejects a previous generation completing after replacement without discarding the new generation cache', async () => { it('rejects a previous generation completing after replacement without discarding the new generation cache', async () => {
const api = await setup(); const api = await setup();
const { instanceId } = await native(api); const { instanceId } = await native(api);
+3 -3
View File
@@ -16,10 +16,10 @@ describe('host detail instances', () => {
db().prepare(`INSERT INTO native_instances (id, server, host_id, generation, library_id, image, port, created_at, retired_at) db().prepare(`INSERT INTO native_instances (id, server, host_id, generation, library_id, image, port, created_at, retired_at)
VALUES (?, ?, ?, 1, 'library', 'runtime:test', 28960, ?, ?)`) VALUES (?, ?, ?, 1, 'library', 'runtime:test', 28960, ?, ?)`)
.run(name, name, hostId, now, retired ? now : null); .run(name, name, hostId, now, retired ? now : null);
db().prepare(`INSERT INTO servers (name, komodo_name, db().prepare(`INSERT INTO servers (name,
container_state, container_checked_at, last_seen_at, last_players, destroyed_at) container_state, container_checked_at, last_seen_at, last_players, destroyed_at)
VALUES (?, ?, 'running', ?, ?, 2, ?)`) VALUES (?, 'running', ?, ?, 2, ?)`)
.run(name, name, now, now, destroyed ? now : null); .run(name, now, now, destroyed ? now : null);
}; };
add('running', host.id); add('running', host.id);
add('unavailable', host.id); add('unavailable', host.id);
+5 -5
View File
@@ -1,7 +1,6 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { newFixture, type Fixture } from '../helpers/fixture.js'; import { newFixture, type Fixture } from '../helpers/fixture.js';
vi.mock('../../src/lib/server/integrations/komodo.js', () => ({ komodoInspectHealthy: vi.fn() }));
let fixture: Fixture; let fixture: Fixture;
let at: number; let at: number;
beforeEach(async () => { beforeEach(async () => {
@@ -15,8 +14,9 @@ afterEach(async () => { vi.useRealTimers(); await fixture.cleanup(); });
async function setup(count = 5, hostCount = 1) { async function setup(count = 5, hostCount = 1) {
const hosts = await import('../../src/lib/server/hosts/index.js'); const hosts = await import('../../src/lib/server/hosts/index.js');
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js'); vi.spyOn(await import('../../src/lib/server/backends/native.js'), 'nativeHealth');
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: true, state: 'running' }); const { nativeHealth } = await import('../../src/lib/server/backends/native.js');
vi.mocked(nativeHealth).mockResolvedValue({ ok: true, state: 'running' });
const hostIds: string[] = []; const hostIds: string[] = [];
for (let h = 0; h < hostCount; h++) { for (let h = 0; h < hostCount; h++) {
const host = hosts.createHost({ name: `Host ${h}` }); hostIds.push(host.id); const host = hosts.createHost({ name: `Host ${h}` }); hostIds.push(host.id);
@@ -26,7 +26,7 @@ async function setup(count = 5, hostCount = 1) {
hosts.enrollHost(hosts.createEnrollment(host.id).token, inventory); hosts.enrollHost(hosts.createEnrollment(host.id).token, inventory);
for (let i = 0; i < count; i++) { for (let i = 0; i < count; i++) {
const name = `server-${h}-${i}`; const name = `server-${h}-${i}`;
db().prepare('INSERT INTO servers(name,komodo_name,created_at) VALUES(?,?,?)').run(name, '', new Date(at - 1000).toISOString()); db().prepare('INSERT INTO servers(name,created_at) VALUES(?,?)').run(name, new Date(at - 1000).toISOString());
db().prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at) db().prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at)
VALUES(?,?,?,'library','image',?,?)`).run(`instance-${h}-${i}`, name, host.id, 28960 + i, new Date(at - 1000).toISOString()); VALUES(?,?,?,'library','image',?,?)`).run(`instance-${h}-${i}`, name, host.id, 28960 + i, new Date(at - 1000).toISOString());
} }
@@ -41,7 +41,7 @@ async function setup(count = 5, hostCount = 1) {
? { ok: true, results: targets.map(target => ({ ...target, reply: { ok: true, text: 'map: mp_kino\nnum score ping guid name\n--- ----- ---- ---- ----\n' } })) } ? { ok: true, results: targets.map(target => ({ ...target, reply: { ok: true, text: 'map: mp_kino\nnum score ping guid name\n--- ----- ---- ---- ----\n' } })) }
: { ok: true, text: 'map: mp_kino\nnum score ping guid name\n--- ----- ---- ---- ----\n' } }); : { ok: true, text: 'map: mp_kino\nnum score ping guid name\n--- ----- ---- ---- ----\n' } });
}); });
return { ...hosts, ...health, ...worker, getServer, db: db(), inspect: vi.mocked(komodoInspectHealthy), hostIds, operations, complete }; return { ...hosts, ...health, ...worker, getServer, db: db(), inspect: vi.mocked(nativeHealth), hostIds, operations, complete };
} }
describe('host status observations', () => { describe('host status observations', () => {
-106
View File
@@ -1,106 +0,0 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { komodoInspectHealthy } from '../../src/lib/server/integrations/komodo.js';
vi.mock('../../src/lib/server/config.js', () => ({
config: {
komodo: { address: 'http://komodo.test', apiKey: 'test', apiSecret: 'test' },
httpTimeoutMs: 1000
},
komodoConfigured: () => true
}));
vi.mock('../../src/lib/server/fleet/servers.js', () => ({
getServer: () => ({ komodo_type: 'deployment', komodo_name: 'saved-resource' })
}));
const fetchMock = vi.fn<typeof fetch>();
beforeEach(() => {
fetchMock.mockReset();
vi.stubGlobal('fetch', fetchMock);
});
afterEach(() => {
vi.unstubAllGlobals();
vi.useRealTimers();
});
describe('Komodo health endpoint fallback', () => {
it('limits an unreachable probe to one timeout and preserves its diagnosis', async () => {
vi.useFakeTimers();
fetchMock.mockImplementation((_url, options) => new Promise((_resolve, reject) => {
options?.signal?.addEventListener('abort', () => reject(new DOMException('aborted', 'AbortError')));
}));
const result = komodoInspectHealthy('server');
await vi.advanceTimersByTimeAsync(1000);
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(await result).toMatchObject({
ok: false,
reason: 'timed out after 1000ms dialing http://komodo.test/read/GetDeploymentContainer'
});
});
it('preserves the network cause without probing alternate endpoints', async () => {
fetchMock.mockRejectedValue(new TypeError('fetch failed', {
cause: Object.assign(new Error('connection refused'), { code: 'ECONNREFUSED' })
}));
expect(await komodoInspectHealthy('server')).toMatchObject({
ok: false,
reason: expect.stringContaining('ECONNREFUSED: connection refused')
});
expect(fetchMock).toHaveBeenCalledTimes(1);
});
it.each([400, 401, 403, 429, 500, 502, 503])('stops on HTTP %i without hiding the response', async status => {
fetchMock.mockResolvedValue(new Response('request failed', { status }));
expect(await komodoInspectHealthy('server')).toEqual({
ok: false, reason: `HTTP ${status}`, snippet: 'request failed'
});
expect(fetchMock).toHaveBeenCalledTimes(1);
});
it.each([
[404, 'Not Found'],
[400, 'unknown variant `GetDeploymentContainer`'],
[405, 'unsupported method'],
[500, 'unknown method'],
[501, 'method not found']
])('tries a compatible endpoint after HTTP %i: %s', async (status, message) => {
fetchMock
.mockResolvedValueOnce(new Response(message as string, { status: status as number }))
.mockResolvedValueOnce(Response.json({ State: { Status: 'running' } }));
expect(await komodoInspectHealthy('server')).toEqual({ ok: true, state: 'running' });
expect(fetchMock.mock.calls.map(([url]) => url)).toEqual([
'http://komodo.test/read/GetDeploymentContainer',
'http://komodo.test/read/InspectDeploymentContainer'
]);
});
it('continues past successful responses with an unknown shape', async () => {
fetchMock
.mockResolvedValueOnce(Response.json({ config: { image: 'server' } }))
.mockResolvedValueOnce(Response.json({ State: { Status: 'running' } }));
expect(await komodoInspectHealthy('server')).toEqual({ ok: true, state: 'running' });
expect(fetchMock).toHaveBeenCalledTimes(2);
});
it('does not overwrite an unhealthy container state with a later response', async () => {
fetchMock.mockResolvedValueOnce(Response.json({ State: { Status: 'exited' } }));
expect(await komodoInspectHealthy('server')).toMatchObject({ ok: false, state: 'exited', reason: 'container state: exited' });
expect(fetchMock).toHaveBeenCalledTimes(1);
});
it('preserves malformed response diagnostics', async () => {
fetchMock.mockResolvedValueOnce(new Response('not json'));
expect(await komodoInspectHealthy('server')).toMatchObject({
ok: false, reason: expect.stringContaining('unparseable json:'), snippet: 'not json'
});
expect(fetchMock).toHaveBeenCalledTimes(1);
});
});
-36
View File
@@ -1,36 +0,0 @@
import { afterEach, beforeEach, expect, it, vi } from 'vitest';
vi.mock('../../src/lib/server/fleet/servers.js', () => ({ getServer: vi.fn() }));
beforeEach(() => {
vi.resetModules(); vi.clearAllMocks();
vi.stubEnv('KOMODO_ADDRESS', 'http://komodo.test');
vi.stubEnv('KOMODO_API_KEY', 'test'); vi.stubEnv('KOMODO_API_SECRET', 'test');
vi.stubGlobal('fetch', vi.fn(async () => Response.json({ success: true })));
});
afterEach(() => { vi.unstubAllGlobals(); vi.unstubAllEnvs(); });
it.each(['start', 'stop'] as const)('sends %s to the saved resource for each Komodo type', async action => {
const { getServer } = await import('../../src/lib/server/fleet/servers.js');
const { komodoRuntimeAction } = await import('../../src/lib/server/integrations/komodo.js');
const verb = action === 'start' ? 'Start' : 'Stop';
for (const [type, suffix, body] of [
['deployment', 'Deployment', { deployment: 'saved-resource' }],
['stack', 'Stack', { stack: 'saved-resource' }],
['container', 'Container', { server: 'host', container: 'saved-resource' }]
] as const) {
vi.mocked(getServer).mockReturnValue({ komodo_type: type, komodo_name: 'saved-resource', komodo_server: 'host' } as ReturnType<typeof getServer>);
expect(await komodoRuntimeAction('ec-server-id', action)).toEqual({ ok: true });
expect(fetch).toHaveBeenLastCalledWith(`http://komodo.test/execute/${verb}${suffix}`, expect.objectContaining({ method: 'POST', body: JSON.stringify(body) }));
}
});
it('rejects missing resource identity and surfaces a rejected stop response', async () => {
const { getServer } = await import('../../src/lib/server/fleet/servers.js');
const { komodoRuntimeAction } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(getServer).mockReturnValue(undefined);
expect(await komodoRuntimeAction('missing', 'stop')).toMatchObject({ ok: false, kind: 'no-name' });
expect(fetch).not.toHaveBeenCalled();
vi.mocked(getServer).mockReturnValue({ komodo_type: 'deployment', komodo_name: 'saved' } as ReturnType<typeof getServer>);
vi.mocked(fetch).mockResolvedValueOnce(Response.json({ success: false, message: 'rejected' }));
expect(await komodoRuntimeAction('server', 'stop')).toMatchObject({ ok: false, message: 'rejected' });
});
-88
View File
@@ -1,88 +0,0 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { komodoEnsureTag, komodoUpdateTagsOnResource } from '../../src/lib/server/integrations/komodo.js';
vi.mock('../../src/lib/server/config.js', () => ({
config: {
komodo: { address: 'http://komodo.test', apiKey: 'test', apiSecret: 'test' },
httpTimeoutMs: 1000
},
komodoConfigured: () => true
}));
const fetchMock = vi.fn<typeof fetch>();
const gameId = '6aa1fb6bc27e735cc0767f5a';
const mapId = '6aabe8567a227ba7d9aad3e9';
const tags = [
{ _id: { $oid: gameId }, name: 't5', owner: '6aa1e0ba91f87b7712e8caf8', color: 'Slate' },
{ _id: { $oid: mapId }, name: 'zombie_cod5_sumpf', owner: '6aa1e0ba91f87b7712e8caf8', color: 'Slate' }
];
beforeEach(() => {
fetchMock.mockReset();
vi.stubGlobal('fetch', fetchMock);
vi.spyOn(console, 'warn').mockImplementation(() => {});
});
afterEach(() => {
vi.unstubAllGlobals();
vi.restoreAllMocks();
});
describe('Komodo tags', () => {
it('reuses the tags from the logged Mongo response and sends string IDs to the deployment', async () => {
fetchMock
.mockResolvedValueOnce(Response.json(tags))
.mockResolvedValueOnce(Response.json(tags))
.mockResolvedValueOnce(Response.json({}));
const game = await komodoEnsureTag('t5');
const map = await komodoEnsureTag('zombie_cod5_sumpf');
expect(game).toEqual({ ok: true, id: gameId });
expect(map).toEqual({ ok: true, id: mapId });
if (!game.ok || !map.ok) throw new Error('Expected both tags to resolve');
expect(await komodoUpdateTagsOnResource('Deployment', 't5-sumpf-1', [game.id, map.id]))
.toEqual({ ok: true });
expect(fetchMock.mock.calls.map(([url]) => url)).toEqual([
'http://komodo.test/read/ListTags',
'http://komodo.test/read/ListTags',
'http://komodo.test/write/UpdateResourceMeta'
]);
expect(JSON.parse(fetchMock.mock.calls[2][1]!.body as string)).toEqual({
target: { type: 'Deployment', id: 't5-sumpf-1' }, tags: [gameId, mapId]
});
expect(console.warn).not.toHaveBeenCalled();
});
it.each([{ id: gameId }, { _id: gameId }])('preserves string ID support: %j', async (id) => {
fetchMock.mockResolvedValueOnce(Response.json([{ ...id, name: 't5' }]));
expect(await komodoEnsureTag('t5')).toEqual({ ok: true, id: gameId });
expect(fetchMock).toHaveBeenCalledTimes(1);
});
it('reads an Extended JSON ID from CreateTag without another lookup', async () => {
fetchMock
.mockResolvedValueOnce(Response.json([]))
.mockResolvedValueOnce(Response.json([]))
.mockResolvedValueOnce(Response.json(tags[0]));
expect(await komodoEnsureTag('t5')).toEqual({ ok: true, id: gameId });
expect(fetchMock).toHaveBeenCalledTimes(3);
expect(fetchMock.mock.calls[2][0]).toBe('http://komodo.test/write/CreateTag');
});
it('recovers when another caller creates the tag after the initial lookup', async () => {
fetchMock
.mockResolvedValueOnce(Response.json([]))
.mockResolvedValueOnce(Response.json([]))
.mockResolvedValueOnce(Response.json({ error: 'E11000 duplicate key error' }, { status: 500 }))
.mockResolvedValueOnce(Response.json(tags));
expect(await komodoEnsureTag('t5')).toEqual({ ok: true, id: gameId });
expect(fetchMock).toHaveBeenCalledTimes(4);
});
it.each([null, {}, { $oid: '' }, { $oid: 123 }])('rejects malformed Mongo IDs: %j', async (_id) => {
fetchMock
.mockResolvedValueOnce(Response.json([{ name: 't5', _id }]))
.mockResolvedValueOnce(Response.json({ error: 'create failed' }, { status: 400 }));
expect(await komodoEnsureTag('t5')).toMatchObject({ ok: false });
});
});
-29
View File
@@ -1,29 +0,0 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
beforeEach(() => {
vi.resetModules();
vi.stubEnv('KOMODO_ADDRESS', 'http://komodo.test');
vi.stubEnv('KOMODO_API_KEY', 'test-key');
vi.stubEnv('KOMODO_API_SECRET', 'test-secret');
});
afterEach(() => { vi.unstubAllGlobals(); vi.unstubAllEnvs(); });
describe('Komodo startup response classification', () => {
it.each([403, 502])('preserves HTTP %i so callers can distinguish rejection from uncertain startup', async httpStatus => {
vi.stubGlobal('fetch', vi.fn(async () => new Response('Deployment response failed', { status: httpStatus })));
const { komodoDeploy } = await import('../../src/lib/server/integrations/komodo.js');
expect(await komodoDeploy('t5_zombie_theater_1')).toMatchObject({ ok: false, kind: 'http-error', httpStatus });
});
it('identifies an explicit success=false response as a confirmed HTTP 200 failure', async () => {
vi.stubGlobal('fetch', vi.fn(async () => Response.json({ success: false, message: 'Image rejected' })));
const { komodoDeploy } = await import('../../src/lib/server/integrations/komodo.js');
expect(await komodoDeploy('t5_zombie_theater_1')).toMatchObject({ ok: false, kind: 'http-error', httpStatus: 200, message: 'Image rejected' });
});
it('returns a structured unreachable result when the startup response times out', async () => {
vi.stubGlobal('fetch', vi.fn(async () => { throw new DOMException('Request aborted', 'AbortError'); }));
const { komodoDeploy } = await import('../../src/lib/server/integrations/komodo.js');
expect(await komodoDeploy('t5_zombie_theater_1')).toMatchObject({ ok: false, kind: 'unreachable', message: expect.stringContaining('timed out') });
});
});
-148
View File
@@ -1,148 +0,0 @@
import { describe, expect, it } from 'vitest';
import { extractUdpPort, extractContainerState } from '../../src/lib/server/integrations/komodo.js';
describe('container readiness', () => {
it.each([
['Up 35 seconds (health: starting)', 'starting'],
['Up about a minute (healthy)', 'running'],
['Up 5 minutes (unhealthy)', 'unhealthy'],
['Up 2 hours', 'running'],
['Exited (1) 5 seconds ago', 'exited']
])('reads nested Komodo container status: %s', (status, expected) => {
expect(extractContainerState({ state: 'Running', container: { state: 'running', status } })).toBe(expected);
});
it('reads lowercase inspect health and does not trust missing containers or configuration', () => {
expect(extractContainerState({ state: { status: 'running', running: true, health: { status: 'starting' } } })).toBe('starting');
expect(extractContainerState({ state: 'Running', container: null })).toBe('unknown');
expect(extractContainerState({ config: { state: 'running' }, status: 'ok' })).toBeNull();
expect(extractContainerState({ state: 'Exited', container: { status: 'Up 1 minute (healthy)' } })).toBe('exited');
});
it.each(['starting', 'unhealthy'])('does not accept a running container whose health is %s', (health) => {
expect(extractContainerState({ State: { Status: 'running', Running: true, Health: { Status: health } } })).toBe(health);
expect(extractContainerState({ state: 'running', health })).toBe(health);
expect(extractContainerState({ state: 'running', health: { status: health } })).toBe(health);
});
it('accepts healthy running containers and preserves stopped states', () => {
expect(extractContainerState({ State: { Status: 'running', Running: true, Health: { Status: 'healthy' } } })).toBe('running');
expect(extractContainerState({ State: { Status: 'exited', Running: false, Health: { Status: 'healthy' } } })).toBe('exited');
});
});
describe('extractUdpPort', () => {
it('returns the published port from docker-compose-style object entries', () => {
const payload = {
state: {
container: {
ports: [
{ protocol: 'tcp', published: 3000, target: 3000 },
{ protocol: 'udp', published: 4976, target: 28960 }
]
}
}
};
expect(extractUdpPort(payload)).toBe(4976);
});
it('parses docker-run-style "host:container/proto" strings', () => {
const payload = { ports: ['3074:3074/tcp', '4977:28960/udp'] };
expect(extractUdpPort(payload)).toBe(4977);
});
it('picks the UDP entry even when TCP appears first', () => {
const payload = {
deeply: {
nested: {
ports: [
{ Protocol: 'tcp', PublishedPort: 80 },
{ Protocol: 'udp', PublishedPort: 5555 }
]
}
}
};
expect(extractUdpPort(payload)).toBe(5555);
});
it('returns null when no UDP mapping exists', () => {
const payload = { ports: [{ protocol: 'tcp', published: 3000 }] };
expect(extractUdpPort(payload)).toBeNull();
});
it('survives cyclic references', () => {
const cyc: Record<string, unknown> = {};
cyc.self = cyc;
expect(extractUdpPort(cyc)).toBeNull();
});
it('recognizes HostPort / hostPort / host_port on object entries', () => {
expect(extractUdpPort({ ports: [{ protocol: 'udp', HostPort: 28962 }] })).toBe(28962);
expect(extractUdpPort({ ports: [{ protocol: 'udp', hostPort: 28963 }] })).toBe(28963);
expect(extractUdpPort({ ports: [{ protocol: 'udp', host_port: 28964 }] })).toBe(28964);
});
it('recognizes `type` alongside `protocol` as the protocol field', () => {
expect(extractUdpPort({ ports: [{ type: 'UDP', published: 28965 }] })).toBe(28965);
});
it('parses host-network "container/udp" strings', () => {
expect(extractUdpPort({ ports: ['28966/udp'] })).toBe(28966);
});
it('handles docker-inspect container-keyed PortBindings shape', () => {
const payload = {
NetworkSettings: {
Ports: {
'28960/udp': [{ HostIp: '0.0.0.0', HostPort: '28967' }],
'3074/tcp': [{ HostIp: '0.0.0.0', HostPort: '3074' }]
}
}
};
expect(extractUdpPort(payload)).toBe(28967);
});
it('prefers `ports`-keyed arrays over unrelated arrays discovered deeper', () => {
// The unrelated array happens to contain a matching entry — the walker
// must still prefer the real `ports` list.
const payload = {
labels: [{ protocol: 'udp', published: 9999 }],
state: { container: { ports: [{ protocol: 'udp', published: 28968 }] } }
};
expect(extractUdpPort(payload)).toBe(28968);
});
it('splits Komodo`s newline-delimited string form of `config.ports`', () => {
// Verbatim shape from a real Komodo GetDeployment response — the
// container publishes UDP 28963 and TCP 28963 as a single string.
const payload = {
name: 't5-nacht-1',
config: {
image: { type: 'Image', params: { image: 'ghcr.io/ayymoss/plutainer:latest' } },
network: 'komodo_default',
ports: '28963:28963/udp\n28963:28963/tcp\n'
}
};
expect(extractUdpPort(payload)).toBe(28963);
});
it('handles a single-line string-form ports value', () => {
expect(extractUdpPort({ config: { ports: '28969:28960/udp' } })).toBe(28969);
});
it('handles a host-network string-form ports value (no host prefix)', () => {
expect(extractUdpPort({ config: { ports: '28970/udp\n' } })).toBe(28970);
});
it('ignores TCP-only string-form ports entries', () => {
expect(extractUdpPort({ config: { ports: '80:80/tcp\n443:443/tcp' } })).toBeNull();
});
});
describe('komodoResolvePortDetailed', () => {
it('is importable and has the discriminated-union shape at the type level', async () => {
// Runtime import shim just to prove it's exported. Behavior is
// covered by the fixture-based extractUdpPort tests above plus manual
// end-to-end verification against a live Komodo.
const mod = await import('../../src/lib/server/integrations/komodo.js');
expect(typeof mod.komodoResolvePortDetailed).toBe('function');
expect(typeof mod.komodoResolveGamePort).toBe('function');
});
});
+4 -251
View File
@@ -1,17 +1,9 @@
import { prepareNativeHost } from '../helpers/native-host.js';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { readFile } from 'node:fs/promises'; import { readFile } from 'node:fs/promises';
import { join } from 'node:path'; import { join } from 'node:path';
import { newFixture, type Fixture } from '../helpers/fixture.js'; import { newFixture, type Fixture } from '../helpers/fixture.js';
vi.mock('../../src/lib/server/integrations/komodo.js', () => ({
komodoCreateDeployment: vi.fn(async () => ({ ok: true })),
komodoDeploy: vi.fn(async () => ({ ok: true })),
komodoDestroyDeployment: vi.fn(async () => ({ ok: true })),
komodoDeleteDeployment: vi.fn(async () => ({ ok: true })),
komodoInspectHealthy: vi.fn(async () => ({ ok: true, state: 'running' })),
komodoEnsureTag: vi.fn(async (name: string) => ({ ok: true, id: name })),
komodoUpdateTagsOnResource: vi.fn(async () => ({ ok: true }))
}));
vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ vi.mock('../../src/lib/server/integrations/rcon.js', () => ({
rconGetPlayerCount: vi.fn(async () => 0), rconGetPlayerCount: vi.fn(async () => 0),
rconPlutoniumVersion: vi.fn(async () => ({ revision: 5354, error: null })), rconPlutoniumVersion: vi.fn(async () => ({ revision: 5354, error: null })),
@@ -27,11 +19,9 @@ const input = { ...selection, gameMode: 'zombie' as const, plutoKey: 'private-pl
beforeEach(async () => { beforeEach(async () => {
vi.resetModules(); vi.clearAllMocks(); vi.resetModules(); vi.clearAllMocks();
fixture = await newFixture([]); fixture = await newFixture([]);
await prepareNativeHost();
vi.stubGlobal('fetch', vi.fn(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354, files: [] }))); vi.stubGlobal('fetch', vi.fn(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354, files: [] })));
vi.stubEnv('KOMODO_ADDRESS', 'http://komodo.test');
vi.stubEnv('KOMODO_API_KEY', 'private-api-key');
vi.stubEnv('KOMODO_API_SECRET', 'private-api-secret');
vi.stubEnv('KOMODO_TARGET_SERVER', 'test-host');
vi.stubEnv('RCON_PASSWORD', 'private-rcon-password'); vi.stubEnv('RCON_PASSWORD', 'private-rcon-password');
}); });
afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); vi.unstubAllGlobals(); }); afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); vi.unstubAllGlobals(); });
@@ -46,42 +36,11 @@ async function seedArtifacts() {
} }
describe('launch recipes and previews', () => { describe('launch recipes and previews', () => {
it('keeps detail polling independent of retained plugin bytes after launching', async () => {
const { createScript } = await import('../../src/lib/server/fleet/scripts.js');
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
const plugin = createScript({ name: 'Large utility plugin', kind: 'plugin', scopeType: 'game', scopeValue: 't5',
targetRelPath: 'large.dll', content: Buffer.alloc(8 * 1024 ** 2, 42) });
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(plugin.version.id, plugin.script.id);
const { queueCreateServer, processCreations } = await import('../../src/lib/server/fleet/create.js');
const { getDeployment } = await import('../../src/lib/server/fleet/deployments.js');
const { recipeForDeployment, summarizeRecipe } = await import('../../src/lib/server/fleet/launch-recipes.js');
const queued = queueCreateServer(input);
if (!queued.ok) throw new Error(queued.message);
const queuedSummary = getDeployment(queued.deploymentId)!.recipe!;
await processCreations();
const full = recipeForDeployment(queued.deploymentId)!.recipe;
const expected = summarizeRecipe(full);
expect(expected.fingerprint).not.toBe(queuedSummary.fingerprint);
expect(full.files.find(file => file.path === 'large.dll')?.content).toBe(plugin.version.content.toString('base64'));
closeDb();
const parse = JSON.parse;
const parsing = vi.spyOn(JSON, 'parse').mockImplementation((value, reviver) => {
if (value.length > 1024 * 1024) throw new Error('Detail polling parsed plugin bytes.');
return parse(value, reviver);
});
try {
for (let read = 0; read < 3; read++) expect(getDeployment(queued.deploymentId)?.recipe).toEqual(expected);
const row = db().prepare('SELECT length(recipe_summary) AS bytes FROM server_launch_recipes WHERE deployment_id = ?')
.get(queued.deploymentId) as { bytes: number };
expect(row.bytes).toBeLessThan(10_000);
} finally { parsing.mockRestore(); }
});
it('previews without reserving capacity, saving keys, exposing bytes, or calling Komodo', async () => { it('previews without reserving capacity, saving keys, exposing bytes, or submitting host commands', async () => {
await seedArtifacts(); await seedArtifacts();
const { previewCreation } = await import('../../src/lib/server/fleet/create.js'); const { previewCreation } = await import('../../src/lib/server/fleet/create.js');
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const { komodoCreateDeployment } = await import('../../src/lib/server/integrations/komodo.js');
const first = previewCreation(selection); const first = previewCreation(selection);
const second = previewCreation(selection); const second = previewCreation(selection);
expect(first).toMatchObject({ name: 't5_zombie_theater_1', port: 28960, hasSavedKey: false, ready: true }); expect(first).toMatchObject({ name: 't5_zombie_theater_1', port: 28960, hasSavedKey: false, ready: true });
@@ -91,68 +50,9 @@ describe('launch recipes and previews', () => {
expect(first.recipe.files[0]).not.toHaveProperty('content'); expect(first.recipe.files[0]).not.toHaveProperty('content');
expect(db().prepare('SELECT * FROM server_keys').all()).toEqual([]); expect(db().prepare('SELECT * FROM server_keys').all()).toEqual([]);
expect(db().prepare('SELECT * FROM server_creation_jobs').all()).toEqual([]); expect(db().prepare('SELECT * FROM server_creation_jobs').all()).toEqual([]);
expect(komodoCreateDeployment).not.toHaveBeenCalled();
expect(JSON.stringify(first)).not.toContain('private-'); expect(JSON.stringify(first)).not.toContain('private-');
}); });
it('uses promotions made after review and queueing, including plugins and composed configuration', async () => {
const { baseline, script } = await seedArtifacts();
const { previewCreation, queueCreateServer, processCreations } = await import('../../src/lib/server/fleet/create.js');
const { createScript, uploadNewVersion } = await import('../../src/lib/server/fleet/scripts.js');
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
const { getDeployment } = await import('../../src/lib/server/fleet/deployments.js');
const { getServerScript } = await import('../../src/lib/server/fleet/inventory.js');
const { recipeForDeployment } = await import('../../src/lib/server/fleet/launch-recipes.js');
const plugin = createScript({ name: 'Game utilities', kind: 'plugin', scopeType: 'game', scopeValue: 't5', targetRelPath: 'utilities.dll', content: Buffer.from('old plugin') });
const overlay = createScript({ name: 'Kino settings', kind: 'config', scopeType: 'map', scopeValue: 'zombie_theater', content: Buffer.from('set g_speed "180"\n') });
for (const item of [plugin, overlay]) db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(item.version.id, item.script.id);
const preview = previewCreation(selection);
const newScript = uploadNewVersion(script.script.id, Buffer.from('main(){ new_perks(); }'));
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(newScript.id, script.script.id);
const newOverlay = uploadNewVersion(overlay.script.id, Buffer.from('set g_speed "200"\n'));
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(newOverlay.id, overlay.script.id);
uploadNewVersion(script.script.id, Buffer.from('main(){ unpromoted_draft(); }'));
const queued = queueCreateServer({ ...input, previewId: preview.id });
if (!queued.ok) throw new Error(queued.message);
expect(recipeForDeployment(queued.deploymentId)?.recipe.files).toContainEqual(expect.objectContaining({ path: 'perks.gsc', versionId: newScript.id }));
const newBaseline = uploadNewVersion(baseline.script.id, Buffer.from('set sv_voice "0"\n'));
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(newBaseline.id, baseline.script.id);
const newPlugin = uploadNewVersion(plugin.script.id, Buffer.from('new plugin'));
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(newPlugin.id, plugin.script.id);
uploadNewVersion(plugin.script.id, Buffer.from('unpromoted plugin'));
closeDb();
await processCreations();
expect(await readFile(join(fixture.t5, queued.name, 'runtime/plutonium/storage/t5/raw/scripts/sp/perks.gsc'), 'utf8')).toBe('main(){ new_perks(); }');
const deployedConfig = await readFile(join(fixture.t5, queued.name, 'configs/dedicated_sp.cfg'), 'utf8');
expect(deployedConfig).toContain('set sv_voice "0"');
expect(deployedConfig).toContain('set g_speed "200"');
expect(await readFile(join(fixture.t5, queued.name, 'runtime/plutonium/plugins/utilities.dll'), 'utf8')).toBe('new plugin');
expect(getDeployment(queued.deploymentId)?.recipe?.files).toContainEqual(expect.objectContaining({ path: 'utilities.dll', versionId: newPlugin.id }));
expect(getServerScript(queued.name, 'perks.gsc')).toMatchObject({ script_version_id: newScript.id, sha256: newScript.sha256 });
expect(getServerScript(queued.name, 'utilities.dll')).toMatchObject({ script_version_id: newPlugin.id, sha256: newPlugin.sha256 });
const recordedConfig = recipeForDeployment(queued.deploymentId)?.recipe.files.find(file => file.path === 'dedicated_sp.cfg');
if (recordedConfig?.content === undefined) throw new Error('Attempted launch must retain the configuration bytes.');
expect(Buffer.from(recordedConfig.content, 'base64').toString()).toBe(deployedConfig);
expect(getDeployment(queued.deploymentId)?.recipe?.fingerprint).not.toBe(preview.recipe.fingerprint);
expect(getDeployment(queued.deploymentId)?.deployment).toMatchObject({ state: 'awaiting-health' });
await processCreations();
expect(getDeployment(queued.deploymentId)?.deployment).toMatchObject({ state: 'done' });
expect(JSON.stringify(getDeployment(queued.deploymentId))).not.toContain('private-');
});
it('uses the latest promotion at provisioning for API clients without a preview', async () => {
const { script } = await seedArtifacts();
const { queueCreateServer, processCreations } = await import('../../src/lib/server/fleet/create.js');
const { uploadNewVersion } = await import('../../src/lib/server/fleet/scripts.js');
const { db } = await import('../../src/lib/server/db/index.js');
const queued = queueCreateServer(input);
if (!queued.ok) throw new Error(queued.message);
const changed = uploadNewVersion(script.script.id, Buffer.from('changed'));
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(changed.id, script.script.id);
await processCreations();
expect(await readFile(join(fixture.t5, queued.name, 'runtime/plutonium/storage/t5/raw/scripts/sp/perks.gsc'), 'utf8')).toBe('changed');
});
it('returns the same operation for repeated submission, even after expiry and restart', async () => { it('returns the same operation for repeated submission, even after expiry and restart', async () => {
const { previewCreation, queueCreateServer } = await import('../../src/lib/server/fleet/create.js'); const { previewCreation, queueCreateServer } = await import('../../src/lib/server/fleet/create.js');
const { db, closeDb } = await import('../../src/lib/server/db/index.js'); const { db, closeDb } = await import('../../src/lib/server/db/index.js');
@@ -167,31 +67,6 @@ describe('launch recipes and previews', () => {
expect(getServerKey(preview.name!)).toBe('private-pluto-key'); expect(getServerKey(preview.name!)).toBe('private-pluto-key');
}); });
it('rejects stale slots and ports before saving a key or queueing a second server', async () => {
const { previewCreation, queueCreateServer } = await import('../../src/lib/server/fleet/create.js');
const { getServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
const preview = previewCreation(selection);
expect(queueCreateServer({ ...input, mapSlug: 'moon' }).ok).toBe(true);
expect(queueCreateServer({ ...input, previewId: preview.id })).toMatchObject({ ok: false, kind: 'no-slot' });
expect(getServerKey(preview.name!)).toBeUndefined();
const fresh = previewCreation(selection);
expect(fresh.port).toBe(28961);
expect(queueCreateServer({ ...input, previewId: fresh.id }).ok).toBe(true);
expect(queueCreateServer({ ...input, previewId: preview.id })).toMatchObject({ ok: false, kind: 'no-slot' });
expect(getServerKey('t5_zombie_theater_2')).toBeUndefined();
});
it('does not provision an already accepted preview again through internal callers', async () => {
const { previewCreation, createServer } = await import('../../src/lib/server/fleet/create.js');
const { komodoCreateDeployment, komodoDeploy } = await import('../../src/lib/server/integrations/komodo.js');
const preview = previewCreation(selection);
const first = await createServer({ ...input, previewId: preview.id });
expect(first.ok).toBe(true);
expect(await createServer({ ...input, previewId: preview.id })).toEqual(first);
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
expect(komodoDeploy).toHaveBeenCalledTimes(1);
});
it('carries preview identity through the public creation API and rejects malformed requests', async () => { it('carries preview identity through the public creation API and rejects malformed requests', async () => {
const { POST: preview } = await import('../../src/routes/api/deployments/preview/+server.js'); const { POST: preview } = await import('../../src/routes/api/deployments/preview/+server.js');
const { POST: launch } = await import('../../src/routes/api/servers/+server.js'); const { POST: launch } = await import('../../src/routes/api/servers/+server.js');
@@ -218,93 +93,6 @@ describe('launch recipes and previews', () => {
expect(db().prepare('SELECT * FROM server_keys').all()).toEqual([]); expect(db().prepare('SELECT * FROM server_keys').all()).toEqual([]);
}); });
it('reports configuration blockers while still allowing recipes to be saved', async () => {
vi.stubEnv('KOMODO_TARGET_SERVER', ''); vi.stubEnv('RCON_PASSWORD', '');
const { previewCreation, queueCreateServer } = await import('../../src/lib/server/fleet/create.js');
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
const preview = previewCreation(selection);
expect(preview.ready).toBe(false);
expect(preview.checks.filter(c => c.state === 'blocked')).toHaveLength(2);
expect(saveBlueprint({ name: 'For later', previewId: preview.id }).name).toBe('For later');
expect(queueCreateServer({ ...input, previewId: preview.id })).toMatchObject({ ok: false, kind: 'unconfigured' });
});
it('resolves current file eligibility after a blueprint review and queued launch', async () => {
const { script } = await seedArtifacts();
const { previewCreation, queueCreateServer, processCreations } = await import('../../src/lib/server/fleet/create.js');
const { saveBlueprint, deleteBlueprint, listBlueprints, recipeForDeployment } = await import('../../src/lib/server/fleet/launch-recipes.js');
const { createScript, setScope } = await import('../../src/lib/server/fleet/scripts.js');
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
const removed = createScript({ name: 'Removed script', targetRelPath: 'removed.gsc', scopeType: 'game', scopeValue: 't5', content: Buffer.from('removed') });
const unpromoted = createScript({ name: 'Unpromoted script', targetRelPath: 'unpromoted.gsc', scopeType: 'game', scopeValue: 't5', content: Buffer.from('unpromoted') });
for (const item of [removed, unpromoted]) db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(item.version.id, item.script.id);
const blueprint = saveBlueprint({ name: 'Classic night', description: 'Kino lineup', previewId: previewCreation(selection).id });
const stored = db().prepare('SELECT recipe FROM server_blueprints WHERE id = ?').get(blueprint.id) as { recipe: string };
expect(JSON.parse(stored.recipe)).toEqual({ format: 3, entries: [{ count: 1, ...selection }] });
closeDb();
expect(listBlueprints()[0]).toEqual(blueprint);
const preview = previewCreation({ blueprintId: blueprint.id });
deleteBlueprint(blueprint.id);
const queued = queueCreateServer({ ...input, previewId: preview.id });
if (!queued.ok) throw new Error(queued.message);
db().prepare('UPDATE scripts SET current_version_id = NULL WHERE id = ?').run(removed.script.id);
db().prepare('DELETE FROM scripts WHERE id = ?').run(removed.script.id);
db().prepare('UPDATE scripts SET current_version_id = NULL WHERE id = ?').run(unpromoted.script.id);
setScope({ scriptId: script.script.id, scopeType: 'map', scopeValue: 'zombie_moon' });
const added = createScript({ name: 'New telemetry', targetRelPath: 'telemetry.gsc', scopeType: 'map', scopeValue: 'zombie_theater', content: Buffer.from('newly promoted telemetry') });
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(added.version.id, added.script.id);
createScript({ name: 'Draft telemetry', targetRelPath: 'draft.gsc', scopeType: 'game', scopeValue: 't5', content: Buffer.from('never promoted') });
await processCreations();
for (const file of ['perks.gsc', 'removed.gsc', 'unpromoted.gsc', 'draft.gsc']) {
await expect(readFile(join(fixture.t5, queued.name, 'runtime/plutonium/storage/t5/raw/scripts/sp', file))).rejects.toMatchObject({ code: 'ENOENT' });
}
expect(await readFile(join(fixture.t5, queued.name, 'runtime/plutonium/storage/t5/raw/scripts/sp/telemetry.gsc'), 'utf8')).toBe('newly promoted telemetry');
expect(recipeForDeployment(queued.deploymentId)?.blueprintName).toBe('Classic night');
expect(recipeForDeployment(queued.deploymentId)?.recipe.files.filter(file => file.kind === 'script')).toEqual([
expect.objectContaining({ path: 'telemetry.gsc', versionId: added.version.id })
]);
});
it('launches again with current promotions while preserving the completed deployment record', async () => {
const { script } = await seedArtifacts();
const { previewCreation, createServer, queueCreateServer, processCreations } = await import('../../src/lib/server/fleet/create.js');
const { saveBlueprint, recipeForDeployment } = await import('../../src/lib/server/fleet/launch-recipes.js');
const { uploadNewVersion } = await import('../../src/lib/server/fleet/scripts.js');
const { db } = await import('../../src/lib/server/db/index.js');
const original = await createServer(input);
if (!original.ok) throw new Error(original.message);
const historicalRecipe = recipeForDeployment(original.deploymentId);
const promoted = uploadNewVersion(script.script.id, Buffer.from('current telemetry'));
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(promoted.id, script.script.id);
const replay = previewCreation({ deploymentId: original.deploymentId });
expect(replay).toMatchObject({ name: 't5_zombie_theater_2', port: 28961, hasSavedKey: false });
expect(replay.recipe.files).toContainEqual(expect.objectContaining({ path: 'perks.gsc', versionId: promoted.id }));
const blueprint = saveBlueprint({ name: 'Restored', deploymentId: original.deploymentId });
expect(blueprint.entries).toEqual([{ count: 1, game: 't5', mapSlug: 'theater', mapName: 'Kino Der Toten' }]);
const stored = db().prepare('SELECT recipe FROM server_blueprints WHERE id = ?').get(blueprint.id) as { recipe: string };
expect(JSON.parse(stored.recipe)).toEqual({ format: 3, entries: [{ count: 1, ...selection }] });
expect(previewCreation({ blueprintId: blueprint.id }).recipe.files).toContainEqual(expect.objectContaining({ path: 'perks.gsc', versionId: promoted.id }));
const queued = queueCreateServer({ ...input, previewId: replay.id });
if (!queued.ok) throw new Error(queued.message);
await processCreations();
expect(await readFile(join(fixture.t5, queued.name, 'runtime/plutonium/storage/t5/raw/scripts/sp/perks.gsc'), 'utf8')).toBe('current telemetry');
expect(recipeForDeployment(original.deploymentId)).toEqual(historicalRecipe);
expect(await readFile(join(fixture.t5, original.name, 'runtime/plutonium/storage/t5/raw/scripts/sp/perks.gsc'), 'utf8')).toBe('main(){ old_perks(); }');
});
it('preserves the exact T6 map variant in a blueprint', async () => {
const { previewCreation, queueCreateServer, processCreations } = await import('../../src/lib/server/fleet/create.js');
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
const preview = previewCreation({ game: 't6', mapSlug: 'town' });
const blueprint = saveBlueprint({ name: 'Town survival', previewId: preview.id });
const saved = previewCreation({ blueprintId: blueprint.id });
expect(saved.recipe).toMatchObject({ game: 't6', mapSlug: 'town', expectedMap: 'zm_transit', configFile: 'dedicated_zm.cfg' });
const queued = queueCreateServer({ ...input, game: 't6', mapSlug: 'town', previewId: saved.id });
if (!queued.ok) throw new Error(queued.message);
await processCreations();
expect(await readFile(join(fixture.t6, queued.name, 'configs/electric_cherry_map.cfg'), 'utf8')).toContain('execgts zm_standard_town.cfg map zm_transit');
});
it('validates blueprint names, sources, and case-insensitive duplicates', async () => { it('validates blueprint names, sources, and case-insensitive duplicates', async () => {
const { previewCreation } = await import('../../src/lib/server/fleet/create.js'); const { previewCreation } = await import('../../src/lib/server/fleet/create.js');
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js'); const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
@@ -316,38 +104,3 @@ describe('launch recipes and previews', () => {
expect(() => saveBlueprint({ name: ' classic night ', previewId })).toThrow(/already exists/); expect(() => saveBlueprint({ name: ' classic night ', previewId })).toThrow(/already exists/);
}); });
}); });
describe('resuming verification', () => {
it('resumes an interrupted creation without creating or restarting its container again', async () => {
const { createServer, processCreations } = await import('../../src/lib/server/fleet/create.js');
const { db } = await import('../../src/lib/server/db/index.js');
const { interruptDeployments, getDeployment, resumeVerification } = await import('../../src/lib/server/fleet/deployments.js');
const { komodoCreateDeployment, komodoDeploy } = await import('../../src/lib/server/integrations/komodo.js');
const created = await createServer(input);
if (!created.ok) throw new Error(created.message);
db().prepare("UPDATE server_deployments SET state = 'running' WHERE id = ?").run(created.deploymentId);
interruptDeployments();
expect(getDeployment(created.deploymentId)?.canResumeVerification).toBe(true);
expect(resumeVerification(created.deploymentId)).toBe(true);
expect(resumeVerification(created.deploymentId)).toBe(false);
await processCreations();
expect(getDeployment(created.deploymentId)?.deployment).toMatchObject({ state: 'done', error: null });
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
expect(komodoDeploy).toHaveBeenCalledTimes(1);
});
it('refuses to observe a replacement server or interfere with a later destruction', async () => {
const { createServer, destroyServer } = await import('../../src/lib/server/fleet/create.js');
const { db } = await import('../../src/lib/server/db/index.js');
const { finishDeployment, resumeVerification, getDeployment } = await import('../../src/lib/server/fleet/deployments.js');
const created = await createServer(input);
if (!created.ok) throw new Error(created.message);
finishDeployment(created.deploymentId, { ok: false, message: 'Interrupted' });
db().prepare('UPDATE servers SET game_port = game_port + 1 WHERE name = ?').run(created.name);
expect(resumeVerification(created.deploymentId)).toBe(false);
db().prepare('UPDATE servers SET game_port = ? WHERE name = ?').run(created.port, created.name);
expect(getDeployment(created.deploymentId)?.canResumeVerification).toBe(true);
destroyServer(created.name);
expect(resumeVerification(created.deploymentId)).toBe(false);
});
});
+2 -2
View File
@@ -122,8 +122,8 @@ it('migrates legacy participation as unknown before snapshot backfill and preser
seed(handle, 1, { status: 'active' }); seed(handle, 1, { status: 'active' });
handle.exec('ALTER TABLE leaderboard_match_players DROP COLUMN first_observed_round'); handle.exec('ALTER TABLE leaderboard_match_players DROP COLUMN first_observed_round');
handle.exec('ALTER TABLE leaderboard_match_players DROP COLUMN present_at_start'); handle.exec('ALTER TABLE leaderboard_match_players DROP COLUMN present_at_start');
handle.prepare(`INSERT INTO servers (name, komodo_name, game, map) handle.prepare(`INSERT INTO servers (name, game, map)
VALUES ('t5_zombie_moon_1', 'moon', 't5', 'zombie_moon')`).run(); VALUES ('t5_zombie_moon_1', 't5', 'zombie_moon')`).run();
const saved: GameTelemetry = { event: 'game_end', server: 't5_zombie_moon_1', map: 'zombie_moon', round: 8, const saved: GameTelemetry = { event: 'game_end', server: 't5_zombie_moon_1', map: 'zombie_moon', round: 8,
players: [player('202', 80)], receivedAt: timestamp(2) }; players: [player('202', 80)], receivedAt: timestamp(2) };
handle.prepare('INSERT INTO server_telemetry (server, round, players, received_at, payload) VALUES (?, ?, ?, ?, ?)') handle.prepare('INSERT INTO server_telemetry (server, round, players, received_at, payload) VALUES (?, ?, ?, ?, ?)')
+1 -1
View File
@@ -11,7 +11,7 @@ const cfgPath = (server: string) => join(fixture.t5, server, 'configs', 'dedicat
beforeEach(async () => { beforeEach(async () => {
vi.resetModules(); vi.resetModules();
fixture = await newFixture(servers); fixture = await newFixture(servers);
vi.stubEnv('KOMODO_ADDRESS', '');
vi.stubEnv('RCON_PASSWORD', ''); vi.stubEnv('RCON_PASSWORD', '');
const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js'); const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js');
await seedServerRegistry(); await seedServerRegistry();
+3 -3
View File
@@ -15,8 +15,8 @@ beforeEach(async () => {
received = 0; received = 0;
fixture = await newFixture([]); fixture = await newFixture([]);
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
db().prepare("INSERT INTO servers (name, komodo_name, game, map) VALUES (?, ?, 't5', 'zombie_theater')") db().prepare("INSERT INTO servers (name, game, map) VALUES (?, 't5', 'zombie_theater')")
.run(server, server); .run(server);
}); });
afterEach(async () => fixture.cleanup()); afterEach(async () => fixture.cleanup());
@@ -89,7 +89,7 @@ describe('completed match details', () => {
db().prepare('DELETE FROM servers WHERE name = ?').run(server); db().prepare('DELETE FROM servers WHERE name = ?').run(server);
const deleted = getMatchDetail(1)!; const deleted = getMatchDetail(1)!;
expect(deleted).toEqual({ ...before, server_deleted_at: expect.any(String) }); expect(deleted).toEqual({ ...before, server_deleted_at: expect.any(String) });
db().prepare("INSERT INTO servers (name, komodo_name, game, map) VALUES (?, 'replacement', 't6', 'zm_transit')").run(server); db().prepare("INSERT INTO servers (name, game, map) VALUES (?, 't6', 'zm_transit')").run(server);
const { recordLeaderboardTelemetry } = await import('../../src/lib/server/fleet/leaderboard-ingest.js'); const { recordLeaderboardTelemetry } = await import('../../src/lib/server/fleet/leaderboard-ingest.js');
recordLeaderboardTelemetry(db(), { event: 'game_end', server, map: 'zm_transit', round: 20, recordLeaderboardTelemetry(db(), { event: 'game_end', server, map: 'zm_transit', round: 20,
players: [player('999', { kills: 999 })], receivedAt: timestamp(4) }); players: [player('999', { kills: 999 })], receivedAt: timestamp(4) });
+4 -4
View File
@@ -64,8 +64,8 @@ describe('historical metrics', () => {
const at = Date.UTC(2026, 0, 2); const at = Date.UTC(2026, 0, 2);
expect(readMetrics(1, at)).toMatchObject({ firstRetainedSampleAt: null, latestSampleAt: null, expect(readMetrics(1, at)).toMatchObject({ firstRetainedSampleAt: null, latestSampleAt: null,
registeredServerCount: 0, servers: [], maps: [] }); registeredServerCount: 0, servers: [], maps: [] });
db().prepare('INSERT INTO servers (name, komodo_name, destroyed_at) VALUES (?, ?, ?)').run('current', 'current', null); db().prepare('INSERT INTO servers (name, destroyed_at) VALUES (?, ?)').run('current', null);
db().prepare('INSERT INTO servers (name, komodo_name, destroyed_at) VALUES (?, ?, ?)').run('removed', 'removed', new Date(at).toISOString()); db().prepare('INSERT INTO servers (name, destroyed_at) VALUES (?, ?)').run('removed', new Date(at).toISOString());
const sample = { server: 'removed', game: 't6', map: 'Town', players: 1, capacity: null }; const sample = { server: 'removed', game: 't6', map: 'Town', players: 1, capacity: null };
recordMetrics([sample], at); recordMetrics([sample], at);
const emptyRange = readMetrics(1, at + 2 * 86400_000); const emptyRange = readMetrics(1, at + 2 * 86400_000);
@@ -341,8 +341,8 @@ describe('historical metrics', () => {
recordMetrics([sample, { ...sample, server: 'removed' }, { ...sample, server: 'tombstone' }], at); recordMetrics([sample, { ...sample, server: 'removed' }, { ...sample, server: 'tombstone' }], at);
// A newly registered name can link to the current server without asserting // A newly registered name can link to the current server without asserting
// that its earlier historical observations belong to this registration. // that its earlier historical observations belong to this registration.
db().prepare('INSERT INTO servers (name, komodo_name, created_at) VALUES (?, ?, ?)').run('reused', 'reused', new Date(at + 15_000).toISOString()); db().prepare('INSERT INTO servers (name, created_at) VALUES (?, ?)').run('reused', new Date(at + 15_000).toISOString());
db().prepare('INSERT INTO servers (name, komodo_name, destroyed_at) VALUES (?, ?, ?)').run('tombstone', 'tombstone', new Date(at).toISOString()); db().prepare('INSERT INTO servers (name, destroyed_at) VALUES (?, ?)').run('tombstone', new Date(at).toISOString());
const result = readMetrics(1, at + 30_000); const result = readMetrics(1, at + 30_000);
expect(result.servers.find(server => server.server === 'reused')!.currentRegistered).toBe(true); expect(result.servers.find(server => server.server === 'reused')!.currentRegistered).toBe(true);
expect(result.servers.filter(server => server.server !== 'reused').every(server => !server.currentRegistered)).toBe(true); expect(result.servers.filter(server => server.server !== 'reused').every(server => !server.currentRegistered)).toBe(true);
+6 -5
View File
@@ -10,7 +10,7 @@ const server = 't5_zombie_theater_1';
beforeEach(async () => { beforeEach(async () => {
vi.resetModules(); vi.resetModules();
fixture = await newFixture([]); fixture = await newFixture([]);
vi.stubEnv('KOMODO_ADDRESS', ''); vi.stubEnv('KOMODO_API_KEY', ''); vi.stubEnv('KOMODO_API_SECRET', '');
vi.stubEnv('RCON_PASSWORD', ''); vi.stubEnv('RCON_PASSWORD', '');
}); });
afterEach(async () => { vi.restoreAllMocks(); await fixture.cleanup(); vi.unstubAllEnvs(); }); afterEach(async () => { vi.restoreAllMocks(); await fixture.cleanup(); vi.unstubAllEnvs(); });
@@ -30,12 +30,12 @@ async function intent() {
} }
describe('native creation intents before placement', () => { describe('native creation intents before placement', () => {
it('retains its identity and native history across restart without falling into legacy provisioning', async () => { it('retains its identity and native history across restart without changing ownership', async () => {
const id = await intent(); const id = await intent();
const { db, closeDb } = await import('../../src/lib/server/db/index.js'); const { db, closeDb } = await import('../../src/lib/server/db/index.js');
const { getDeployment, interruptDeployments } = await import('../../src/lib/server/provisioning/history.js'); const { getDeployment, interruptDeployments } = await import('../../src/lib/server/provisioning/history.js');
const { nativeDeployment, unplacedNativeCreation } = await import('../../src/lib/server/backends/native.js'); const { nativeDeployment, unplacedNativeCreation } = await import('../../src/lib/server/backends/native.js');
const { nextLobbyNumber, processCreations } = await import('../../src/lib/server/provisioning/instances.js'); const { nextLobbyNumber } = await import('../../src/lib/server/provisioning/instances.js');
const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js'); const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js');
expect(nativeDeployment(id)).toBe(true); expect(nativeDeployment(id)).toBe(true);
expect(unplacedNativeCreation(server)).toBe(id); expect(unplacedNativeCreation(server)).toBe(id);
@@ -43,7 +43,8 @@ describe('native creation intents before placement', () => {
expect(requestLifecycle(server, 'stop')).toMatchObject({ ok: false, status: 409 }); expect(requestLifecycle(server, 'stop')).toMatchObject({ ok: false, status: 409 });
closeDb(); closeDb();
interruptDeployments(); interruptDeployments();
await processCreations(); const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
await processProvisioning();
expect(getDeployment(id)).toMatchObject({ backend: 'native', canCancelCreation: true, agentOperation: null, expect(getDeployment(id)).toMatchObject({ backend: 'native', canCancelCreation: true, agentOperation: null,
deployment: { state: 'queued', server_exists: 0 } }); deployment: { state: 'queued', server_exists: 0 } });
expect(db().prepare('SELECT port FROM server_creation_jobs WHERE deployment_id=?').get(id)).toEqual({ port: null }); expect(db().prepare('SELECT port FROM server_creation_jobs WHERE deployment_id=?').get(id)).toEqual({ port: null });
@@ -78,7 +79,7 @@ describe('native creation intents before placement', () => {
it('will not cancel an unplaced historical intent against a replacement server', async () => { it('will not cancel an unplaced historical intent against a replacement server', async () => {
const id = await intent(); const id = await intent();
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
db().prepare("INSERT INTO servers(name,komodo_name) VALUES(?,'replacement')").run(server); db().prepare("INSERT INTO servers(name) VALUES(?)").run(server);
const { cancelCreation, canCancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js'); const { cancelCreation, canCancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
expect(canCancelCreation(id)).toBe(false); expect(canCancelCreation(id)).toBe(false);
expect(cancelCreation(id)).toBeNull(); expect(cancelCreation(id)).toBeNull();
+2 -2
View File
@@ -22,8 +22,8 @@ async function setup(capable = true) {
}); });
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const database = db(); const database = db();
database.prepare(`INSERT INTO servers(name,komodo_name,game,map,game_mode,map_slug,created_by) database.prepare(`INSERT INTO servers(name,game,map,game_mode,map_slug,created_by)
VALUES(?,'','t5','zombie_theater','zombie','theater','ui')`).run(server); VALUES(?,'t5','zombie_theater','zombie','theater','ui')`).run(server);
database.prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at) database.prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at)
VALUES(?,?,?,'library','image',28960,?)`).run(randomUUID(), server, host.id, new Date().toISOString()); VALUES(?,?,?,'library','image',28960,?)`).run(randomUUID(), server, host.id, new Date().toISOString());
+1 -1
View File
@@ -19,7 +19,7 @@ beforeEach(async () => {
fixture = await newFixture([]); fixture = await newFixture([]);
vi.stubEnv('GAME_FILES_ROOT', join(fixture.root, 'libraries')); vi.stubEnv('GAME_FILES_ROOT', join(fixture.root, 'libraries'));
vi.stubEnv('PLUTAINER_IMAGE', image); vi.stubEnv('RCON_PASSWORD', 'test-password'); vi.stubEnv('PLUTAINER_IMAGE', image); vi.stubEnv('RCON_PASSWORD', 'test-password');
vi.stubEnv('KOMODO_ADDRESS', ''); vi.stubEnv('KOMODO_API_KEY', ''); vi.stubEnv('KOMODO_API_SECRET', '');
const { createHost, createEnrollment, enrollHost, heartbeat } = await import('../../src/lib/server/hosts/index.js'); const { createHost, createEnrollment, enrollHost, heartbeat } = await import('../../src/lib/server/hosts/index.js');
hostId = createHost({ name: 'Preparation test host', portMin: 28960, portMax: 28980 }).id; hostId = createHost({ name: 'Preparation test host', portMin: 28960, portMax: 28980 }).id;
const inventory = { cpuCount: 32, cpuUsagePercent: 0, memoryTotal: 64 * 1024 ** 3, const inventory = { cpuCount: 32, cpuUsagePercent: 0, memoryTotal: 64 * 1024 ** 3,
+1 -1
View File
@@ -10,7 +10,7 @@ async function setup() {
const host = hosts.createHost({ name: 'Observation host' }); const host = hosts.createHost({ name: 'Observation host' });
hosts.enrollHost(hosts.createEnrollment(host.id).token, { protocolVersion: 1 }); hosts.enrollHost(hosts.createEnrollment(host.id).token, { protocolVersion: 1 });
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
db().prepare("INSERT INTO servers(name,komodo_name) VALUES('test','')").run(); db().prepare("INSERT INTO servers(name) VALUES('test')").run();
db().prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at) db().prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at)
VALUES('instance','test',?,'library','image',28960,?)`).run(host.id, new Date().toISOString()); VALUES('instance','test',?,'library','image',28960,?)`).run(host.id, new Date().toISOString());
const native = await import('../../src/lib/server/backends/native.js'); const native = await import('../../src/lib/server/backends/native.js');
+6 -7
View File
@@ -19,7 +19,7 @@ beforeEach(async () => {
fixture = await newFixture([]); fixture = await newFixture([]);
vi.stubEnv('GAME_FILES_ROOT', join(fixture.root, 'libraries')); vi.stubEnv('GAME_FILES_ROOT', join(fixture.root, 'libraries'));
vi.stubEnv('PLUTAINER_IMAGE', image); vi.stubEnv('RCON_PASSWORD', 'test-password'); vi.stubEnv('PLUTAINER_IMAGE', image); vi.stubEnv('RCON_PASSWORD', 'test-password');
vi.stubEnv('KOMODO_ADDRESS', ''); vi.stubEnv('KOMODO_API_KEY', ''); vi.stubEnv('KOMODO_API_SECRET', '');
const { createHost, createEnrollment, enrollHost } = await import('../../src/lib/server/hosts/index.js'); const { createHost, createEnrollment, enrollHost } = await import('../../src/lib/server/hosts/index.js');
hostId = createHost({ name: 'Native Debian', portMin: 28960, portMax: 28970 }).id; hostId = createHost({ name: 'Native Debian', portMin: 28960, portMax: 28970 }).id;
enrollHost(createEnrollment(hostId).token, inventory()); enrollHost(createEnrollment(hostId).token, inventory());
@@ -250,7 +250,6 @@ describe('native instance provisioning', () => {
expect(result.progress?.checks[0].source).toBe('agent'); expect(result.progress?.checks[0].source).toBe('agent');
const steps = provisioningGroups(result).flatMap(group => group.steps); const steps = provisioningGroups(result).flatMap(group => group.steps);
expect(steps.every(step => step.state === 'done')).toBe(true); expect(steps.every(step => step.state === 'done')).toBe(true);
expect(steps.some(step => step.label.includes('Komodo'))).toBe(false);
expect(result.agentOperation?.state).toBe('completed'); expect(result.agentOperation?.state).toBe('completed');
}); });
@@ -410,7 +409,7 @@ describe('native instance provisioning', () => {
expect(nativeInstance(job.name)?.library_id).toBe(acceptedId); expect(nativeInstance(job.name)?.library_id).toBe(acceptedId);
}); });
it.each(['t5', 't6'] as const)('queues and verifies a %s deployment without Komodo', async game => { it.each(['t5', 't6'] as const)('queues and verifies a %s deployment through its host agent', async game => {
const job = await queue(game); await advance(); const job = await queue(game); await advance();
const operation = await operationFor(job.deploymentId); const operation = await operationFor(job.deploymentId);
expect(job.port).toBeNull(); expect(job.port).toBeNull();
@@ -498,8 +497,8 @@ describe('native instance provisioning', () => {
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const { probeVersion } = await import('../../src/lib/server/integrations/plutonium.js'); const { probeVersion } = await import('../../src/lib/server/integrations/plutonium.js');
const actual = await vi.importActual<typeof import('../../src/lib/server/integrations/plutonium.js')>('../../src/lib/server/integrations/plutonium.js'); const actual = await vi.importActual<typeof import('../../src/lib/server/integrations/plutonium.js')>('../../src/lib/server/integrations/plutonium.js');
const komodo = await import('../../src/lib/server/integrations/komodo.js'); const backend = await import('../../src/lib/server/backends/native.js');
vi.spyOn(komodo, 'komodoInspectHealthy').mockResolvedValue({ ok: true, state: 'running' }); vi.spyOn(backend, 'nativeHealth').mockResolvedValue({ ok: true, state: 'running' });
vi.mocked(probeVersion).mockImplementationOnce(actual.probeVersion); vi.mocked(probeVersion).mockImplementationOnce(actual.probeVersion);
// Version evidence must follow the submitted command, even on a fast test host. // Version evidence must follow the submitted command, even on a fast test host.
db().prepare('UPDATE server_deployments SET submitted_at=? WHERE id=?').run(new Date(Date.now() - 1000).toISOString(), rebuild.deploymentId); db().prepare('UPDATE server_deployments SET submitted_at=? WHERE id=?').run(new Date(Date.now() - 1000).toISOString(), rebuild.deploymentId);
@@ -526,8 +525,8 @@ describe('native instance provisioning', () => {
const { probeVersion } = await import('../../src/lib/server/integrations/plutonium.js'); const { probeVersion } = await import('../../src/lib/server/integrations/plutonium.js');
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js'); const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
const actual = await vi.importActual<typeof import('../../src/lib/server/integrations/plutonium.js')>('../../src/lib/server/integrations/plutonium.js'); const actual = await vi.importActual<typeof import('../../src/lib/server/integrations/plutonium.js')>('../../src/lib/server/integrations/plutonium.js');
const komodo = await import('../../src/lib/server/integrations/komodo.js'); const backend = await import('../../src/lib/server/backends/native.js');
vi.spyOn(komodo, 'komodoInspectHealthy').mockResolvedValue({ ok: true, state: 'running' }); vi.spyOn(backend, 'nativeHealth').mockResolvedValue({ ok: true, state: 'running' });
const submittedAt = new Date(Date.now() - 1000).toISOString(); const submittedAt = new Date(Date.now() - 1000).toISOString();
db().prepare('UPDATE server_deployments SET submitted_at=? WHERE id=?').run(submittedAt, rebuild.deploymentId); db().prepare('UPDATE server_deployments SET submitted_at=? WHERE id=?').run(submittedAt, rebuild.deploymentId);
db().prepare('UPDATE restart_rollout SET requested_at=? WHERE operation_id=?').run(submittedAt, rebuild.deploymentId); db().prepare('UPDATE restart_rollout SET requested_at=? WHERE operation_id=?').run(submittedAt, rebuild.deploymentId);
@@ -0,0 +1,51 @@
import Database from 'better-sqlite3';
import { readFile } from 'node:fs/promises';
import { afterEach, beforeEach, expect, it, vi } from 'vitest';
import { newFixture, type Fixture } from '../helpers/fixture.js';
let fixture: Fixture;
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
afterEach(async () => { await fixture.cleanup(); });
it('retires only obsolete backend columns while preserving instances, history, keys and foreign keys', async () => {
const schema = await readFile(new URL('../../src/lib/server/db/schema.sql', import.meta.url), 'utf8');
const previous = new Database(fixture.dbPath);
// Model the former required resource identity and an unrelated extension.
previous.exec(schema);
previous.exec(`ALTER TABLE servers ADD COLUMN komodo_type TEXT NOT NULL DEFAULT 'deployment';
ALTER TABLE servers ADD COLUMN komodo_name TEXT NOT NULL;
ALTER TABLE servers ADD COLUMN komodo_server TEXT;
ALTER TABLE servers ADD COLUMN operator_note TEXT;`);
previous.prepare(`INSERT INTO servers(name,komodo_name,komodo_server,game,game_port,operator_note)
VALUES('native-server','','old-host','t5',28960,'keep this')`).run();
previous.prepare("INSERT INTO server_keys(server_id,pluto_key) VALUES('native-server','saved-key')").run();
previous.prepare(`INSERT INTO server_deployments(id,server,action,state,stage,started_at,triggered_by)
VALUES('creation','native-server','create','done','Ready','2026-01-01','fixture')`).run();
previous.prepare(`INSERT INTO server_deployment_events(deployment_id,created_at,message)
VALUES('creation','2026-01-01','Original receipt')`).run();
const { migrateNativeInstances } = await import('../../src/lib/server/backends/schema.js');
migrateNativeInstances(previous);
previous.prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at)
VALUES('instance','native-server','host','library','image',28960,'2026-01-01')`).run();
previous.close();
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
for (let pass = 0; pass < 2; pass++) {
const handle = db();
const columns = (handle.prepare('PRAGMA table_info(servers)').all() as { name: string }[]).map(column => column.name);
expect(columns).not.toEqual(expect.arrayContaining(['komodo_type']));
expect(columns).not.toContain('komodo_name');
expect(columns).not.toContain('komodo_server');
expect(columns).toContain('operator_note');
expect(handle.prepare('SELECT name,game_port,operator_note FROM servers WHERE name=?').get('native-server'))
.toEqual({ name: 'native-server', game_port: 28960, operator_note: 'keep this' });
expect(handle.prepare('SELECT id,generation FROM native_instances').all()).toEqual([{ id: 'instance', generation: 1 }]);
expect(handle.prepare('SELECT pluto_key FROM server_keys').get()).toEqual({ pluto_key: 'saved-key' });
expect(handle.prepare('SELECT message FROM server_deployment_events').get()).toEqual({ message: 'Original receipt' });
expect(handle.pragma('foreign_key_check')).toEqual([]);
expect(handle.pragma('foreign_keys', { simple: true })).toBe(1);
closeDb();
}
// Native inserts must work without supplying the retired required field.
expect(() => db().prepare("INSERT INTO servers(name) VALUES('new-native-server')").run()).not.toThrow();
});
+57 -4
View File
@@ -13,14 +13,14 @@ const server = 't5_zombie_theater_1';
const healthy = { exists: true, healthy: true, state: 'running' }; const healthy = { exists: true, healthy: true, state: 'running' };
beforeEach(async () => { beforeEach(async () => {
vi.resetModules(); vi.clearAllMocks(); fixture = await newFixture([]); vi.resetModules(); vi.clearAllMocks(); fixture = await newFixture([]);
vi.stubEnv('KOMODO_ADDRESS', ''); vi.stubEnv('KOMODO_API_KEY', ''); vi.stubEnv('KOMODO_API_SECRET', '');
const { createHost, createEnrollment, enrollHost } = await import('../../src/lib/server/hosts/index.js'); const { createHost, createEnrollment, enrollHost } = await import('../../src/lib/server/hosts/index.js');
hostId = createHost({ name: 'Safety test host' }).id; hostId = createHost({ name: 'Safety test host' }).id;
enrollHost(createEnrollment(hostId).token, { protocolVersion: 1, dockerRunning: true, directoriesWritable: true, enrollHost(createEnrollment(hostId).token, { protocolVersion: 1, dockerRunning: true, directoriesWritable: true,
storageAvailable: 100_000_000_000, runtimeImages: ['plutainer:test'], libraries: [{ id: 'published-game-version', state: 'ready' }] }); storageAvailable: 100_000_000_000, runtimeImages: ['plutainer:test'], libraries: [{ id: 'published-game-version', state: 'ready' }] });
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
instanceId = randomUUID(); instanceId = randomUUID();
db().prepare("INSERT INTO servers(name,komodo_name,game,map,game_mode,map_slug,game_port,container_state) VALUES(?,'','t5','zombie_theater','zombie','theater',28960,'running')").run(server); db().prepare("INSERT INTO servers(name,game,map,game_mode,map_slug,game_port,container_state) VALUES(?,'t5','zombie_theater','zombie','theater',28960,'running')").run(server);
db().prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at) VALUES(?,?,?,'published-game-version','plutainer:test',28960,?)`).run(instanceId, server, hostId, new Date().toISOString()); db().prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at) VALUES(?,?,?,'published-game-version','plutainer:test',28960,?)`).run(instanceId, server, hostId, new Date().toISOString());
const backend = await import('../../src/lib/server/backends/native.js'); const backend = await import('../../src/lib/server/backends/native.js');
vi.spyOn(backend, 'nativeRequest').mockImplementation(async (_instance, kind) => { vi.spyOn(backend, 'nativeRequest').mockImplementation(async (_instance, kind) => {
@@ -31,8 +31,8 @@ beforeEach(async () => {
afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); vi.restoreAllMocks(); }); afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); vi.restoreAllMocks(); });
async function advance() { async function advance() {
const { processNativeProvisioning } = await import('../../src/lib/server/provisioning/native.js'); const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
await processNativeProvisioning(() => null); await processProvisioning();
} }
async function operationId(id: string) { async function operationId(id: string) {
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
@@ -119,3 +119,56 @@ describe('native disruption safety at dispatch', () => {
expect(db().prepare("SELECT COUNT(*) AS count FROM ec_host_operations WHERE kind='instance.restart'").get()).toEqual({ count: 0 }); expect(db().prepare("SELECT COUNT(*) AS count FROM ec_host_operations WHERE kind='instance.restart'").get()).toEqual({ count: 0 });
}); });
}); });
describe('native scheduler ownership', () => {
it('rejects lifecycle commands for a registry row without a host instance', async () => {
const { db } = await import('../../src/lib/server/db/index.js');
const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js');
const { destroyServer } = await import('../../src/lib/server/provisioning/instances.js');
db().prepare("INSERT INTO servers(name) VALUES('unplaced-registry')").run();
expect(requestLifecycle('unplaced-registry', 'restart')).toMatchObject({ ok: false, status: 503 });
expect(destroyServer('unplaced-registry')).toMatchObject({ ok: false, kind: 'conflict' });
expect(db().prepare("SELECT id FROM server_deployments WHERE server='unplaced-registry'").all()).toEqual([]);
});
it('shares restart capacity until an agent receipt and fresh health release the first slot', async () => {
const { db } = await import('../../src/lib/server/db/index.js');
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js');
const { reportHostOperation } = await import('../../src/lib/server/hosts/index.js');
saveFleetSettings({ ...fleetSettings(), restartConcurrency: 1 });
db().prepare("INSERT INTO servers(name,game,map,game_port) VALUES('second','t5','zombie_theater',28961)").run();
db().prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at)
SELECT 'second-instance','second',host_id,library_id,image,28961,created_at FROM native_instances WHERE id=?`).run(instanceId);
const first = await waitingRestart();
const second = requestLifecycle('second', 'restart');
if (!second.ok) throw new Error(second.error);
await advance();
const command = await operationId(first);
expect(command).toEqual(expect.any(String));
expect(await operationId(second.deploymentId)).toBeNull();
await advance();
expect(await operationId(first)).toBe(command);
expect(await operationId(second.deploymentId)).toBeNull();
reportHostOperation(hostId, command!, { state: 'completed', result: healthy });
await advance();
await advance();
expect(db().prepare('SELECT state FROM server_deployments WHERE id=?').get(first)).toEqual({ state: 'done' });
expect(await operationId(second.deploymentId)).toEqual(expect.any(String));
expect(db().prepare("SELECT id FROM ec_host_operations WHERE kind='instance.restart'").all()).toHaveLength(2);
});
it('holds a paused campaign and cancels withdrawn work before delivering a restart', async () => {
const id = await stageCampaign();
const { db } = await import('../../src/lib/server/db/index.js');
db().prepare("UPDATE deployment_servers SET restart_state='held' WHERE deployment_id='campaign-race'").run();
await advance();
expect(await operationId(id)).toBeNull();
expect(db().prepare('SELECT state FROM server_deployments WHERE id=?').get(id)).toEqual({ state: 'waiting' });
db().prepare("UPDATE deployment_servers SET restart_state='done' WHERE deployment_id='campaign-race'").run();
await advance();
expect(await operationId(id)).toBeNull();
expect(db().prepare('SELECT state FROM server_deployments WHERE id=?').get(id)).toEqual({ state: 'cancelled' });
expect(db().prepare("SELECT id FROM ec_host_operations WHERE kind='instance.restart'").all()).toEqual([]);
});
});
+2 -2
View File
@@ -10,7 +10,7 @@ describe('persistent activity', () => {
it('keeps beta failures and retries across reopening without duplicating polls', async () => { it('keeps beta failures and retries across reopening without duplicating polls', async () => {
const { db, closeDb } = await import('../../src/lib/server/db/index.js'); const { db, closeDb } = await import('../../src/lib/server/db/index.js');
const handle = db(); const handle = db();
handle.prepare("INSERT INTO servers (name, komodo_name) VALUES ('server', 'server')").run(); handle.prepare("INSERT INTO servers (name) VALUES ('server')").run();
handle.prepare("INSERT INTO scripts (id, name, target_rel_path) VALUES (1, 'test', 'test.gsc')").run(); handle.prepare("INSERT INTO scripts (id, name, target_rel_path) VALUES (1, 'test', 'test.gsc')").run();
handle.prepare("INSERT INTO script_versions (id, script_id, version, sha256, size, content) VALUES (1, 1, 1, 'hash', 0, X'')").run(); handle.prepare("INSERT INTO script_versions (id, script_id, version, sha256, size, content) VALUES (1, 1, 1, 'hash', 0, X'')").run();
handle.prepare("INSERT INTO betas (id, script_id, script_version_id, trigger_kind, state) VALUES (1, 1, 1, 'ui', 'running')").run(); handle.prepare("INSERT INTO betas (id, script_id, script_version_id, trigger_kind, state) VALUES (1, 1, 1, 'ui', 'running')").run();
@@ -29,7 +29,7 @@ describe('persistent activity', () => {
it('records campaign files, restart and health milestones, and cascades cleanup', async () => { it('records campaign files, restart and health milestones, and cascades cleanup', async () => {
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const handle = db(); const handle = db();
handle.prepare("INSERT INTO servers (name, komodo_name) VALUES ('server', 'server')").run(); handle.prepare("INSERT INTO servers (name) VALUES ('server')").run();
handle.prepare("INSERT INTO deployments (id, action, started_at) VALUES ('campaign', 'promote', '2026-09-17T00:00:00Z')").run(); handle.prepare("INSERT INTO deployments (id, action, started_at) VALUES ('campaign', 'promote', '2026-09-17T00:00:00Z')").run();
handle.prepare("INSERT INTO deployment_files (deployment_id, server, verb, rel_path) VALUES ('campaign', 'server', 'updated', 'test.gsc')").run(); handle.prepare("INSERT INTO deployment_files (deployment_id, server, verb, rel_path) VALUES ('campaign', 'server', 'updated', 'test.gsc')").run();
handle.prepare("INSERT INTO deployment_servers (deployment_id, server, files_changed, restart_state) VALUES ('campaign', 'server', 1, 'pending')").run(); handle.prepare("INSERT INTO deployment_servers (deployment_id, server, files_changed, restart_state) VALUES ('campaign', 'server', 1, 'pending')").run();
+1 -1
View File
@@ -5,7 +5,7 @@ let fixture: Fixture;
beforeEach(async () => { beforeEach(async () => {
vi.resetModules(); vi.resetModules();
fixture = await newFixture(); fixture = await newFixture();
delete process.env.KOMODO_ADDRESS;
delete process.env.RCON_PASSWORD; delete process.env.RCON_PASSWORD;
}); });
afterEach(async () => { await fixture.cleanup(); }); afterEach(async () => { await fixture.cleanup(); });
+22 -236
View File
@@ -3,7 +3,6 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { newFixture, type Fixture } from '../helpers/fixture.js'; import { newFixture, type Fixture } from '../helpers/fixture.js';
import { maintenanceTimeInZone, nextMaintenanceAt, parsePlutoniumManifest, parsePlutoniumRevision, PLUTONIUM_MANIFEST_URL } from '../../src/lib/plutonium.js'; import { maintenanceTimeInZone, nextMaintenanceAt, parsePlutoniumManifest, parsePlutoniumRevision, PLUTONIUM_MANIFEST_URL } from '../../src/lib/plutonium.js';
vi.mock('../../src/lib/server/integrations/komodo.js', () => ({ komodoRestart: vi.fn(), komodoInspectHealthy: vi.fn() }));
vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconGetPlayerCount: vi.fn(), rconStatusResult: vi.fn(), rconPlutoniumVersion: vi.fn() })); vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconGetPlayerCount: vi.fn(), rconStatusResult: vi.fn(), rconPlutoniumVersion: vi.fn() }));
let fixture: Fixture; let fixture: Fixture;
@@ -13,15 +12,15 @@ beforeEach(async () => {
vi.useFakeTimers({ toFake: ['Date'] }); vi.useFakeTimers({ toFake: ['Date'] });
vi.setSystemTime(new Date('2026-09-21T05:59:00Z')); vi.setSystemTime(new Date('2026-09-21T05:59:00Z'));
fixture = await newFixture(servers); fixture = await newFixture(servers);
vi.stubEnv('KOMODO_ADDRESS', 'http://komodo.test');
vi.stubEnv('KOMODO_API_KEY', 'test'); vi.stubEnv('KOMODO_API_SECRET', 'test');
vi.stubEnv('RCON_PASSWORD', 'test'); vi.stubEnv('DEPLOYMENT_RESTART_CONCURRENCY', '1'); vi.stubEnv('RCON_PASSWORD', 'test'); vi.stubEnv('DEPLOYMENT_RESTART_CONCURRENCY', '1');
vi.stubGlobal('fetch', vi.fn().mockImplementation(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354, files: [] }))); vi.stubGlobal('fetch', vi.fn().mockImplementation(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354, files: [] })));
const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js'); const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js');
await seedServerRegistry(); await seedServerRegistry();
const k = await import('../../src/lib/server/integrations/komodo.js'); vi.spyOn(await import('../../src/lib/server/backends/native.js'), 'nativeHealth');
vi.mocked(k.komodoRestart).mockResolvedValue({ ok: true }); const k = await import('../../src/lib/server/backends/native.js');
vi.mocked(k.komodoInspectHealthy).mockResolvedValue({ ok: true, state: 'running' });
vi.mocked(k.nativeHealth).mockResolvedValue({ ok: true, state: 'running' });
const r = await import('../../src/lib/server/integrations/rcon.js'); const r = await import('../../src/lib/server/integrations/rcon.js');
vi.mocked(r.rconGetPlayerCount).mockResolvedValue(0); vi.mocked(r.rconGetPlayerCount).mockResolvedValue(0);
vi.mocked(r.rconPlutoniumVersion).mockResolvedValue({ revision: 5354, error: null }); vi.mocked(r.rconPlutoniumVersion).mockResolvedValue({ revision: 5354, error: null });
@@ -38,18 +37,18 @@ describe('Plutonium versions', () => {
const m = await api(); const m = await api();
await m.checkPlutonium(); await m.checkPlutonium();
const { db, closeDb } = await import('../../src/lib/server/db/index.js'); const { db, closeDb } = await import('../../src/lib/server/db/index.js');
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js'); const { nativeHealth } = await import('../../src/lib/server/backends/native.js');
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js'); const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
const saved = db().prepare('SELECT * FROM plutonium_versions WHERE server = ?').get(servers[0]); const saved = db().prepare('SELECT * FROM plutonium_versions WHERE server = ?').get(servers[0]);
db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by) db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by)
VALUES ('busy', ?, ?, ?, 'Provisioning', ?, 'test')`).run(servers[0], action, state, new Date().toISOString()); VALUES ('busy', ?, ?, ?, 'Provisioning', ?, 'test')`).run(servers[0], action, state, new Date().toISOString());
closeDb(); closeDb();
vi.mocked(komodoInspectHealthy).mockClear(); vi.mocked(nativeHealth).mockClear();
vi.mocked(rconPlutoniumVersion).mockClear(); vi.mocked(rconPlutoniumVersion).mockClear();
vi.setSystemTime(Date.now() + 1_000); vi.setSystemTime(Date.now() + 1_000);
await m.checkPlutonium(); await m.checkPlutonium();
expect(komodoInspectHealthy).toHaveBeenCalledTimes(1); expect(nativeHealth).toHaveBeenCalledTimes(1);
expect(komodoInspectHealthy).toHaveBeenCalledWith(servers[1]); expect(nativeHealth).toHaveBeenCalledWith(servers[1]);
expect(rconPlutoniumVersion).toHaveBeenCalledTimes(1); expect(rconPlutoniumVersion).toHaveBeenCalledTimes(1);
expect(rconPlutoniumVersion).toHaveBeenCalledWith(servers[1]); expect(rconPlutoniumVersion).toHaveBeenCalledWith(servers[1]);
expect(db().prepare('SELECT * FROM plutonium_versions WHERE server = ?').get(servers[0])).toEqual(saved); expect(db().prepare('SELECT * FROM plutonium_versions WHERE server = ?').get(servers[0])).toEqual(saved);
@@ -63,17 +62,17 @@ describe('Plutonium versions', () => {
const m = await api(); const m = await api();
await m.checkPlutonium(); await m.checkPlutonium();
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js'); const { nativeHealth } = await import('../../src/lib/server/backends/native.js');
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js'); const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
const saved = db().prepare('SELECT * FROM plutonium_versions WHERE server = ?').get(servers[0]); const saved = db().prepare('SELECT * FROM plutonium_versions WHERE server = ?').get(servers[0]);
db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by) db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by)
VALUES ('failed-restart', ?, 'restart', 'failed', 'Timed out', ?, 'test')`).run(servers[0], new Date().toISOString()); VALUES ('failed-restart', ?, 'restart', 'failed', 'Timed out', ?, 'test')`).run(servers[0], new Date().toISOString());
db().prepare('INSERT INTO restart_rollout (server, requested_at, operation_id) VALUES (?, ?, ?)') db().prepare('INSERT INTO restart_rollout (server, requested_at, operation_id) VALUES (?, ?, ?)')
.run(servers[0], new Date().toISOString(), 'failed-restart'); .run(servers[0], new Date().toISOString(), 'failed-restart');
vi.mocked(komodoInspectHealthy).mockClear(); vi.mocked(nativeHealth).mockClear();
vi.mocked(rconPlutoniumVersion).mockClear(); vi.mocked(rconPlutoniumVersion).mockClear();
await m.checkPlutonium(); await m.checkPlutonium();
expect(komodoInspectHealthy).not.toHaveBeenCalledWith(servers[0]); expect(nativeHealth).not.toHaveBeenCalledWith(servers[0]);
expect(rconPlutoniumVersion).not.toHaveBeenCalledWith(servers[0]); expect(rconPlutoniumVersion).not.toHaveBeenCalledWith(servers[0]);
expect(db().prepare('SELECT * FROM plutonium_versions WHERE server = ?').get(servers[0])).toEqual(saved); expect(db().prepare('SELECT * FROM plutonium_versions WHERE server = ?').get(servers[0])).toEqual(saved);
db().prepare('UPDATE restart_rollout SET healthy_at = ? WHERE server = ?').run(new Date().toISOString(), servers[0]); db().prepare('UPDATE restart_rollout SET healthy_at = ? WHERE server = ?').run(new Date().toISOString(), servers[0]);
@@ -86,13 +85,13 @@ describe('Plutonium versions', () => {
await m.checkPlutonium(); await m.checkPlutonium();
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const { probeVersion } = await import('../../src/lib/server/integrations/plutonium.js'); const { probeVersion } = await import('../../src/lib/server/integrations/plutonium.js');
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js'); const { nativeHealth } = await import('../../src/lib/server/backends/native.js');
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js'); const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
const saved = db().prepare('SELECT * FROM plutonium_versions WHERE server = ?').get(servers[0]); const saved = db().prepare('SELECT * FROM plutonium_versions WHERE server = ?').get(servers[0]);
const queue = () => db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by) const queue = () => db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by)
VALUES ('new-rebuild', ?, 'rebuild', 'waiting', 'Waiting', ?, 'test')`).run(servers[0], new Date().toISOString()); VALUES ('new-rebuild', ?, 'rebuild', 'waiting', 'Waiting', ?, 'test')`).run(servers[0], new Date().toISOString());
vi.mocked(rconPlutoniumVersion).mockClear(); vi.mocked(rconPlutoniumVersion).mockClear();
if (phase === 'health') vi.mocked(komodoInspectHealthy).mockImplementationOnce(async () => { if (phase === 'health') vi.mocked(nativeHealth).mockImplementationOnce(async () => {
queue(); return { ok: true, state: 'running' }; queue(); return { ok: true, state: 'running' };
}); });
else vi.mocked(rconPlutoniumVersion).mockImplementationOnce(async () => { else vi.mocked(rconPlutoniumVersion).mockImplementationOnce(async () => {
@@ -105,7 +104,7 @@ describe('Plutonium versions', () => {
}); });
it('coalesces published-release checks without probing the fleet', async () => { it('coalesces published-release checks without probing the fleet', async () => {
const { checkRelease, release } = await import('../../src/lib/server/integrations/plutonium.js'); const { checkRelease, release } = await import('../../src/lib/server/integrations/plutonium.js');
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js'); const { nativeHealth } = await import('../../src/lib/server/backends/native.js');
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js'); const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
const first = checkRelease(); const first = checkRelease();
const second = checkRelease(); const second = checkRelease();
@@ -113,7 +112,7 @@ describe('Plutonium versions', () => {
await Promise.all([first, second]); await Promise.all([first, second]);
expect(fetch).toHaveBeenCalledTimes(1); expect(fetch).toHaveBeenCalledTimes(1);
expect(release()).toMatchObject({ revision: 5354, error: null }); expect(release()).toMatchObject({ revision: 5354, error: null });
expect(komodoInspectHealthy).not.toHaveBeenCalled(); expect(nativeHealth).not.toHaveBeenCalled();
expect(rconPlutoniumVersion).not.toHaveBeenCalled(); expect(rconPlutoniumVersion).not.toHaveBeenCalled();
await checkRelease(); await checkRelease();
expect(fetch).toHaveBeenCalledTimes(2); expect(fetch).toHaveBeenCalledTimes(2);
@@ -266,7 +265,7 @@ describe('Plutonium versions', () => {
// Both registrations have no lifecycle submission, so only their // Both registrations have no lifecycle submission, so only their
// registry identity distinguishes the old and replacement instances. // registry identity distinguishes the old and replacement instances.
db().prepare('DELETE FROM servers WHERE name = ?').run(servers[0]); db().prepare('DELETE FROM servers WHERE name = ?').run(servers[0]);
db().prepare('INSERT INTO servers (name, komodo_name, created_at) VALUES (?, ?, ?)').run(servers[0], servers[0], now); db().prepare('INSERT INTO servers (name, created_at) VALUES (?, ?)').run(servers[0], now);
} }
vi.setSystemTime(Date.now() + 1); vi.setSystemTime(Date.now() + 1);
expect(await probeVersion(servers[0])).toBe(5354); expect(await probeVersion(servers[0])).toBe(5354);
@@ -314,103 +313,9 @@ describe('maintenance timezones', () => {
expect(maintenanceTimeInZone(schedule, 'America/New_York')).toBe('02:00'); expect(maintenanceTimeInZone(schedule, 'America/New_York')).toBe('02:00');
expect(m.plutoniumStatus().schedule.nextAt).toBe('2026-09-21T06:00:00.000Z'); expect(m.plutoniumStatus().schedule.nextAt).toBe('2026-09-21T06:00:00.000Z');
}); });
it('persists the zone and advances in local calendar days across DST', async () => {
vi.setSystemTime(new Date('2026-03-07T10:59:00Z'));
const m = await api(); await m.checkPlutonium();
m.saveMaintenanceSchedule(true, '06:00', 'America/New_York');
const { closeDb } = await import('../../src/lib/server/db/index.js'); closeDb();
expect(m.plutoniumStatus().schedule.nextAt).toBe('2026-03-07T11:00:00.000Z');
vi.setSystemTime(new Date('2026-03-07T11:00:00Z'));
await maintenanceTick();
expect(m.plutoniumStatus().schedule).toMatchObject({ timeZone: 'America/New_York', nextAt: '2026-03-08T10:00:00.000Z' });
});
}); });
describe('persistent maintenance', () => { describe('persistent maintenance', () => {
it('submits lifecycle work without executing it and rejects cancellation after Provisioning submits', async () => {
const m = await api(); await m.checkPlutonium(); m.queueMaintenance([servers[0]]);
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
await m.maintenanceTick();
expect(komodoRestart).not.toHaveBeenCalled(); expect(rconGetPlayerCount).not.toHaveBeenCalled();
const job = m.plutoniumStatus().jobs[0];
expect(job.provisioning_id).toBeTruthy();
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
await processProvisioning();
// The source has not mirrored submission yet; cancellation must still
// consult the authoritative operation, not its stale queued state.
expect(m.plutoniumStatus().jobs[0].state).toBe('queued');
expect(m.cancelMaintenance(job.id)).toBe(false);
expect(komodoRestart).toHaveBeenCalledTimes(1);
vi.setSystemTime(Date.now() + 15_000); await processProvisioning();
await m.maintenanceTick();
expect(m.plutoniumStatus().jobs[0].state).toBe('done');
});
it('waits for occupied and unreachable servers, deduplicates requests, and verifies after recovery', async () => {
const m = await api(); await m.checkPlutonium();
expect(m.queueMaintenance([servers[0]])).toBe(1);
expect(m.queueMaintenance([servers[0]])).toBe(0);
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(rconGetPlayerCount).mockResolvedValueOnce(2).mockResolvedValueOnce(null);
await maintenanceTick(); await maintenanceTick();
expect(komodoRestart).not.toHaveBeenCalled();
await Promise.all([maintenanceTick(), maintenanceTick()]);
expect(komodoRestart).toHaveBeenCalledTimes(1);
expect(m.plutoniumStatus().jobs[0].state).toBe('restarting');
await maintenanceTick();
expect(m.plutoniumStatus().jobs[0].state).toBe('restarting');
vi.setSystemTime(Date.now() + 15_000);
await maintenanceTick();
expect(m.plutoniumStatus().jobs[0]).toMatchObject({ state: 'done', detail: 'Server recovered; running r5354.' });
});
it('shares rolling capacity across servers and survives closing the database', async () => {
const m = await api(); await m.checkPlutonium(); m.queueMaintenance(servers);
await maintenanceTick();
const { closeDb } = await import('../../src/lib/server/db/index.js'); closeDb();
expect(m.plutoniumStatus().jobs.map(j => j.state)).toEqual(['queued', 'restarting']);
vi.setSystemTime(Date.now() + 15_000); await maintenanceTick();
expect(m.plutoniumStatus().jobs.map(j => j.state)).toEqual(['restarting', 'done']);
});
it('reports a healthy but outdated server as failed without repeatedly restarting it', async () => {
const m = await api(); await m.checkPlutonium(); m.queueMaintenance([servers[0]]);
await maintenanceTick();
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
vi.mocked(rconPlutoniumVersion).mockResolvedValue({ revision: 5300, error: null });
vi.setSystemTime(Date.now() + 15_000); await maintenanceTick(); await maintenanceTick();
expect(m.plutoniumStatus().jobs[0]).toMatchObject({ state: 'failed', target_revision: 5354 });
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
expect(komodoRestart).toHaveBeenCalledTimes(1);
});
it('does not retry an ambiguous restart timeout', async () => {
const m = await api(); await m.checkPlutonium(); m.queueMaintenance([servers[0]]);
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoRestart).mockResolvedValue({ ok: false, kind: 'http-error', message: 'timeout' });
await maintenanceTick(); vi.setSystemTime(Date.now() + 15_000); await maintenanceTick();
expect(m.plutoniumStatus().jobs[0].state).toBe('failed');
expect(komodoRestart).toHaveBeenCalledTimes(1);
});
it('honors cancellation arriving during the player check', async () => {
const m = await api(); await m.checkPlutonium(); m.queueMaintenance([servers[0]]);
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
vi.mocked(rconGetPlayerCount).mockImplementationOnce(async () => {
m.cancelMaintenance(m.plutoniumStatus().jobs[0].id); return 0;
});
await maintenanceTick();
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
expect(komodoRestart).not.toHaveBeenCalled();
expect(m.plutoniumStatus().jobs[0].state).toBe('cancelled');
});
it('holds maintenance while lifecycle work is queued', async () => {
const m = await api(); await m.checkPlutonium(); m.queueMaintenance([servers[0]]);
const { db } = await import('../../src/lib/server/db/index.js');
db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by)
VALUES ('destroy-1', ?, 'destroy', 'queued', 'queued', ?, 'test')`).run(servers[0], new Date().toISOString());
await maintenanceTick();
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
expect(komodoRestart).not.toHaveBeenCalled();
expect(m.plutoniumStatus().jobs[0].detail).toContain('another provisioning operation');
});
it('waits to attach legacy maintenance while another operation owns the server', async () => { it('waits to attach legacy maintenance while another operation owns the server', async () => {
const m = await api(); await m.checkPlutonium(); const m = await api(); await m.checkPlutonium();
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
@@ -425,29 +330,6 @@ describe('persistent maintenance', () => {
expect(m.plutoniumStatus().jobs[0].provisioning_id).toBeTruthy(); expect(m.plutoniumStatus().jobs[0].provisioning_id).toBeTruthy();
expect(db().prepare('SELECT COUNT(*) AS count FROM server_deployments').get()).toEqual({ count: 2 }); expect(db().prepare('SELECT COUNT(*) AS count FROM server_deployments').get()).toEqual({ count: 2 });
}); });
it.each([false, true])('adopts a submitted legacy restart with retained recovery capacity (linked operation: %s)', async linked => {
const m = await api(); await m.checkPlutonium();
const { db } = await import('../../src/lib/server/db/index.js');
const submittedAt = new Date().toISOString();
db().prepare(`INSERT INTO maintenance_jobs (server, requested_at, restarted_at, state)
VALUES (?, ?, ?, 'restarting')`).run(servers[0], submittedAt, submittedAt);
if (linked) {
db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, submitted_at, triggered_by)
VALUES ('legacy-restart', ?, 'restart', 'awaiting-health', 'Waiting for health', ?, ?, 'test')`).run(servers[0], submittedAt, submittedAt);
}
db().prepare('INSERT INTO restart_rollout (server, requested_at, operation_id) VALUES (?, ?, ?)')
.run(servers[0], submittedAt, linked ? 'legacy-restart' : null);
await m.maintenanceTick();
const job = m.plutoniumStatus().jobs[0];
expect(job).toMatchObject({ state: 'restarting', operation_state: 'awaiting-health' });
expect(job.provisioning_id).toBeTruthy();
if (linked) expect(job.provisioning_id).toBe('legacy-restart');
expect(db().prepare('SELECT operation_id FROM restart_rollout WHERE server = ?').get(servers[0])).toEqual({ operation_id: job.provisioning_id });
vi.setSystemTime(Date.now() + 15_000); await maintenanceTick();
expect(m.plutoniumStatus().jobs[0].state).toBe('done');
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
expect(komodoRestart).not.toHaveBeenCalled();
});
it('defers completed maintenance verification while a later operation owns the server, then resumes', async () => { it('defers completed maintenance verification while a later operation owns the server, then resumes', async () => {
const m = await api(); await m.checkPlutonium(); m.queueMaintenance([servers[0]]); const m = await api(); await m.checkPlutonium(); m.queueMaintenance([servers[0]]);
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
@@ -455,12 +337,12 @@ describe('persistent maintenance', () => {
db().prepare("UPDATE server_deployments SET state = 'done' WHERE id = ?").run(job.provisioning_id); db().prepare("UPDATE server_deployments SET state = 'done' WHERE id = ?").run(job.provisioning_id);
db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by) db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by)
VALUES ('later-restart', ?, 'restart', 'waiting', 'Waiting to restart', ?, 'test')`).run(servers[0], new Date().toISOString()); VALUES ('later-restart', ?, 'restart', 'waiting', 'Waiting to restart', ?, 'test')`).run(servers[0], new Date().toISOString());
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js'); const { nativeHealth } = await import('../../src/lib/server/backends/native.js');
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js'); const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
vi.mocked(komodoInspectHealthy).mockClear(); vi.mocked(rconPlutoniumVersion).mockClear(); vi.mocked(nativeHealth).mockClear(); vi.mocked(rconPlutoniumVersion).mockClear();
await m.maintenanceTick(); await m.maintenanceTick();
expect(m.plutoniumStatus().jobs[0]).toMatchObject({ state: 'queued', detail: expect.stringContaining('before verifying') }); expect(m.plutoniumStatus().jobs[0]).toMatchObject({ state: 'queued', detail: expect.stringContaining('before verifying') });
expect(komodoInspectHealthy).not.toHaveBeenCalled(); expect(rconPlutoniumVersion).not.toHaveBeenCalled(); expect(nativeHealth).not.toHaveBeenCalled(); expect(rconPlutoniumVersion).not.toHaveBeenCalled();
db().prepare("UPDATE server_deployments SET state = 'done' WHERE id = 'later-restart'").run(); db().prepare("UPDATE server_deployments SET state = 'done' WHERE id = 'later-restart'").run();
await m.maintenanceTick(); await m.maintenanceTick();
expect(m.plutoniumStatus().jobs[0].state).toBe('done'); expect(m.plutoniumStatus().jobs[0].state).toBe('done');
@@ -493,18 +375,6 @@ describe('persistent maintenance', () => {
await m.maintenanceTick(); await m.maintenanceTick();
expect(m.plutoniumStatus().jobs[0]).toMatchObject({ state: 'failed', detail: 'Restart failed' }); expect(m.plutoniumStatus().jobs[0]).toMatchObject({ state: 'failed', detail: 'Restart failed' });
}); });
it('does not repeat an interrupted restart intent that has no matching rollout slot', async () => {
const m = await api(); await m.checkPlutonium(); m.queueMaintenance([servers[0]]);
const { db } = await import('../../src/lib/server/db/index.js');
db().prepare(`UPDATE maintenance_jobs SET state = 'restarting', restarted_at = requested_at`).run();
await maintenanceTick();
expect(m.plutoniumStatus().jobs[0].state).toBe('restarting');
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
expect(komodoRestart).not.toHaveBeenCalled();
vi.setSystemTime(Date.now() + 15_000); await maintenanceTick();
expect(m.plutoniumStatus().jobs[0].state).toBe('done');
expect(komodoRestart).not.toHaveBeenCalled();
});
it('validates API requests without changing the schedule or queue', async () => { it('validates API requests without changing the schedule or queue', async () => {
const { POST } = await import('../../src/routes/api/plutonium/+server.js'); const { POST } = await import('../../src/routes/api/plutonium/+server.js');
const post = (body: unknown) => POST({ request: new Request('http://localhost/api/plutonium', { const post = (body: unknown) => POST({ request: new Request('http://localhost/api/plutonium', {
@@ -521,19 +391,6 @@ describe('persistent maintenance', () => {
expect(m.plutoniumStatus().schedule.enabled).toBe(true); expect(m.plutoniumStatus().schedule.enabled).toBe(true);
expect(m.plutoniumStatus().schedule).toMatchObject({ timeZone: 'America/New_York', nextAt: '2026-09-21T10:00:00.000Z' }); expect(m.plutoniumStatus().schedule).toMatchObject({ timeZone: 'America/New_York', nextAt: '2026-09-21T10:00:00.000Z' });
}); });
it('runs a daily UTC schedule once and coalesces missed days', async () => {
expect(nextMaintenanceAt('06:00', new Date('2026-09-21T06:00:00Z'))).toBe('2026-09-22T06:00:00.000Z');
expect(() => nextMaintenanceAt('24:00')).toThrow();
const m = await api(); await m.checkPlutonium();
expect(m.plutoniumStatus().schedule.enabled).toBe(false);
m.saveMaintenanceSchedule(true, '06:00', 'UTC');
vi.setSystemTime(new Date('2026-09-24T07:00:00Z'));
await maintenanceTick(); await maintenanceTick();
expect(m.plutoniumStatus().jobs).toHaveLength(2);
expect(m.plutoniumStatus().schedule.nextAt).toBe('2026-09-25T06:00:00.000Z');
m.saveMaintenanceSchedule(false, '06:00', 'UTC');
expect(m.plutoniumStatus().schedule.nextAt).toBeNull();
});
}); });
describe('Maintenance policies and runtime updates', () => { describe('Maintenance policies and runtime updates', () => {
@@ -614,64 +471,6 @@ describe('Maintenance policies and runtime updates', () => {
expect(m.queueUpdates([servers[0]])).toBe(1); expect(m.queueUpdates([servers[0]])).toBe(1);
expect(m.plutoniumStatus().jobs[0]).toMatchObject({ server: servers[0], kind: 'update', origin: 'manual' }); expect(m.plutoniumStatus().jobs[0]).toMatchObject({ server: servers[0], kind: 'update', origin: 'manual' });
}); });
it('holds queued automatic updates when their policy is disabled, but finishes submitted recovery while paused', async () => {
const m = await outdated();
const p = await import('../../src/lib/server/maintenance/policies.js'); p.setMaintenancePolicy('updates', true);
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
m.queueUpdates([servers[0]], true); p.setMaintenancePolicy('updates', false);
await processProvisioning(); expect(komodoRestart).not.toHaveBeenCalled();
p.setMaintenancePolicy('updates', true); p.pauseMaintenance(true);
await processProvisioning(); expect(komodoRestart).not.toHaveBeenCalled();
p.pauseMaintenance(false); await processProvisioning(); expect(komodoRestart).toHaveBeenCalledTimes(1);
p.pauseMaintenance(true); vi.setSystemTime(Date.now() + 15_000);
await processProvisioning(); await m.maintenanceTick();
expect(m.plutoniumStatus().jobs[0]).toMatchObject({ state: 'failed', operation_state: 'done' });
expect(komodoRestart).toHaveBeenCalledTimes(1);
});
it('latches failed update verification across restarts and retention, and supports an explicit retry', async () => {
const m = await outdated();
const p = await import('../../src/lib/server/maintenance/policies.js'); p.setMaintenancePolicy('updates', true);
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
db().prepare("UPDATE servers SET desired_state = 'stopped' WHERE name = ?").run(servers[1]);
await maintenanceTick(); vi.setSystemTime(Date.now() + 15_000); await maintenanceTick();
const failed = m.plutoniumStatus().jobs[0];
expect(failed).toMatchObject({ kind: 'update', state: 'failed', operation_state: 'done' });
closeDb(); vi.setSystemTime(Date.now() + 31 * 86_400_000); await maintenanceTick();
expect(m.plutoniumStatus().jobs).toHaveLength(1);
expect(m.retryMaintenance(failed.id)).toBe(1);
expect(m.plutoniumStatus().jobs).toHaveLength(2);
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
vi.mocked(rconPlutoniumVersion).mockResolvedValue({ revision: 5354, error: null });
await maintenanceTick(); vi.setSystemTime(Date.now() + 15_000); await maintenanceTick();
expect(m.plutoniumStatus().jobs[0]).toMatchObject({ state: 'done', origin: 'manual' });
expect(db().prepare("SELECT 1 FROM server_deployments WHERE action = 'rebuild'").get()).toBeUndefined();
});
it('does not restart intentionally stopped servers, including jobs queued before the intent changed', async () => {
const m = await outdated(); m.queueUpdates([servers[0]]);
const { db } = await import('../../src/lib/server/db/index.js');
db().prepare("UPDATE servers SET desired_state = 'stopped'").run();
await maintenanceTick();
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
expect(komodoRestart).not.toHaveBeenCalled();
expect(m.plutoniumStatus().jobs[0].state).toBe('cancelled');
expect(m.queueUpdates()).toBe(0);
expect(m.plutoniumStatus().servers.every(s => s.update_status === 'stopped')).toBe(true);
});
it('separates scheduled restart success from revision verification and coalesces paused runs', async () => {
const m = await outdated(); m.saveMaintenanceSchedule(true, '06:00', 'UTC');
const p = await import('../../src/lib/server/maintenance/policies.js'); p.pauseMaintenance(true);
vi.setSystemTime(new Date('2026-09-24T07:00:00Z')); await m.maintenanceTick();
expect(m.plutoniumStatus().jobs).toHaveLength(0);
p.pauseMaintenance(false); await maintenanceTick();
expect(m.plutoniumStatus().jobs).toHaveLength(2);
expect(m.plutoniumStatus().jobs.every(j => j.kind === 'schedule' && j.target_revision === null)).toBe(true);
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
vi.mocked(rconPlutoniumVersion).mockResolvedValue({ revision: null, error: 'Unknown reply' });
vi.setSystemTime(Date.now() + 15_000); await maintenanceTick();
expect(m.plutoniumStatus().jobs.find(j => j.server === servers[0])?.state).toBe('done');
expect(m.plutoniumStatus().schedule.nextAt).toBe('2026-09-25T06:00:00.000Z');
});
it('does not treat stale, pre-restart, or unhealthy observations as current runtime evidence', async () => { it('does not treat stale, pre-restart, or unhealthy observations as current runtime evidence', async () => {
const m = await outdated(); const m = await outdated();
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
@@ -679,25 +478,12 @@ describe('Maintenance policies and runtime updates', () => {
expect(m.plutoniumStatus().servers[0].update_status).toBe('unknown'); expect(m.plutoniumStatus().servers[0].update_status).toBe('unknown');
vi.setSystemTime(Date.now() + 3_600_000); vi.setSystemTime(Date.now() + 3_600_000);
expect(() => m.queueUpdates()).toThrow('published revision'); expect(() => m.queueUpdates()).toThrow('published revision');
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js'); const { nativeHealth } = await import('../../src/lib/server/backends/native.js');
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js'); const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
vi.mocked(rconPlutoniumVersion).mockClear(); vi.mocked(rconPlutoniumVersion).mockClear();
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'starting', reason: 'Starting' }); vi.mocked(nativeHealth).mockResolvedValue({ ok: false, state: 'starting', reason: 'Starting' });
await m.checkPlutonium(); await m.checkPlutonium();
expect(rconPlutoniumVersion).not.toHaveBeenCalled(); expect(rconPlutoniumVersion).not.toHaveBeenCalled();
expect(m.plutoniumStatus().servers.every(s => s.update_status === 'unknown')).toBe(true); expect(m.plutoniumStatus().servers.every(s => s.update_status === 'unknown')).toBe(true);
}); });
it('adopts existing maintenance jobs without changing their IDs, schedules, or submitted operations', async () => {
const m = await outdated(); m.saveMaintenanceSchedule(true, '06:00', 'America/New_York');
m.queueMaintenance([servers[0]]); await maintenanceTick();
const before = m.plutoniumStatus();
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
db().exec('ALTER TABLE maintenance_jobs DROP COLUMN kind; ALTER TABLE maintenance_jobs DROP COLUMN origin;'); closeDb();
const after = m.plutoniumStatus();
expect(after.schedule).toEqual(before.schedule);
expect(after.jobs[0]).toMatchObject({ id: before.jobs[0].id, provisioning_id: before.jobs[0].provisioning_id, kind: 'legacy', origin: 'manual' });
vi.setSystemTime(Date.now() + 15_000); await maintenanceTick();
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
expect(komodoRestart).toHaveBeenCalledTimes(1);
});
}); });
+1 -1
View File
@@ -32,7 +32,7 @@ describe('provisioning cancellation', () => {
const { beginDeployment } = await import('../../src/lib/server/provisioning/history.js'); const { beginDeployment } = await import('../../src/lib/server/provisioning/history.js');
const { queueFileChange } = await import('../../src/lib/server/fleet/staged-files.js'); const { queueFileChange } = await import('../../src/lib/server/fleet/staged-files.js');
const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js'); const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
db().prepare("INSERT INTO servers(name,komodo_name) VALUES('one','one'),('two','two')").run(); db().prepare("INSERT INTO servers(name) VALUES('one'),('two')").run();
const id = beginDeployment('one','restart'); const id = beginDeployment('one','restart');
const other = beginDeployment('two','restart'); const other = beginDeployment('two','restart');
db().prepare("INSERT INTO restart_rollout(server,requested_at,operation_id) VALUES('one','2026-10-05',?),('two','2026-10-05',?)").run(id, other); db().prepare("INSERT INTO restart_rollout(server,requested_at,operation_id) VALUES('one','2026-10-05',?),('two','2026-10-05',?)").run(id, other);
+4 -4
View File
@@ -22,9 +22,9 @@ describe('provisioning detail projection', () => {
it('keeps legacy full activity while the page and opt-in API return a bounded snapshot without probing', async () => { it('keeps legacy full activity while the page and opt-in API return a bounded snapshot without probing', async () => {
const history = await import('../../src/lib/server/provisioning/history.js'); const history = await import('../../src/lib/server/provisioning/history.js');
const komodo = await import('../../src/lib/server/integrations/komodo.js'); const backend = await import('../../src/lib/server/backends/native.js');
const rcon = await import('../../src/lib/server/integrations/rcon.js'); const rcon = await import('../../src/lib/server/integrations/rcon.js');
const inspect = vi.spyOn(komodo, 'komodoInspectHealthy'); const inspect = vi.spyOn(backend, 'nativeHealth');
const status = vi.spyOn(rcon, 'rconStatusResult'); const status = vi.spyOn(rcon, 'rconStatusResult');
const id = history.beginDeployment('t5_zombie_theater_1', 'create'); const id = history.beginDeployment('t5_zombie_theater_1', 'create');
for (let index = 0; index < 40; index++) history.recordDeploymentEvent(id, `Recorded activity ${index}`); for (let index = 0; index < 40; index++) history.recordDeploymentEvent(id, `Recorded activity ${index}`);
@@ -73,7 +73,7 @@ describe('provisioning detail projection', () => {
it('correlates reservations by operation ID without assigning a later same-name slot to historical work', async () => { it('correlates reservations by operation ID without assigning a later same-name slot to historical work', async () => {
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const { beginDeployment, getDeployment, finishDeployment } = await import('../../src/lib/server/provisioning/history.js'); const { beginDeployment, getDeployment, finishDeployment } = await import('../../src/lib/server/provisioning/history.js');
db().prepare("INSERT INTO servers (name,komodo_name) VALUES ('server','server')").run(); db().prepare("INSERT INTO servers (name) VALUES ('server')").run();
const old = beginDeployment('server', 'restart'); const old = beginDeployment('server', 'restart');
finishDeployment(old, { ok: false, message: 'Timeout' }); finishDeployment(old, { ok: false, message: 'Timeout' });
const current = beginDeployment('server', 'restart'); const current = beginDeployment('server', 'restart');
@@ -104,7 +104,7 @@ describe('provisioning detail projection', () => {
it('preserves a failure outcome and evidence before an observation-only same-ID resume', async () => { it('preserves a failure outcome and evidence before an observation-only same-ID resume', async () => {
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const history = await import('../../src/lib/server/provisioning/history.js'); const history = await import('../../src/lib/server/provisioning/history.js');
db().prepare("INSERT INTO servers (name,komodo_name,game_port) VALUES ('server','server',28960)").run(); db().prepare("INSERT INTO servers (name,game_port) VALUES ('server',28960)").run();
const id = history.beginDeployment('server', 'create'); const id = history.beginDeployment('server', 'create');
db().prepare("INSERT INTO server_creation_jobs VALUES (?, 'server',28960,'zombie','theater','zombie_theater')").run(id); db().prepare("INSERT INTO server_creation_jobs VALUES (?, 'server',28960,'zombie','theater','zombie_theater')").run(id);
history.deploymentStage(id, 'Waiting for game evidence', { phase: 'verify-game', checks: [ history.deploymentStage(id, 'Waiting for game evidence', { phase: 'verify-game', checks: [
-405
View File
@@ -1,405 +0,0 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { join } from 'node:path';
import { newFixture, type Fixture } from '../helpers/fixture.js';
import { gameEvent } from '../helpers/telemetry.js';
vi.mock('../../src/lib/server/integrations/komodo.js', () => ({
komodoRestart: vi.fn(), komodoRuntimeAction: vi.fn(), komodoInspectHealthy: vi.fn(),
komodoCreateDeployment: vi.fn(), komodoDeploy: vi.fn(), komodoDestroyDeployment: vi.fn(), komodoDeleteDeployment: vi.fn(),
komodoEnsureTag: vi.fn(), komodoUpdateTagsOnResource: vi.fn()
}));
vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconGetPlayerCount: vi.fn(), rconStatusResult: vi.fn(), rconPlutoniumVersion: vi.fn() }));
vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}) }));
let fixture: Fixture;
const server = 't5_zombie_theater_1';
const status = (map = 'zombie_theater') => ({ ok: true as const, endpoint: { host: 'test', port: 28960 }, rawReply: '',
status: { players: 0, map, gametype: 'zom', roster: [] } });
beforeEach(async () => {
vi.resetModules(); vi.resetAllMocks();
vi.useFakeTimers({ toFake: ['Date'] });
vi.setSystemTime(new Date('2026-09-28T12:00:00Z'));
vi.stubGlobal('fetch', vi.fn(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354 })));
fixture = await newFixture([server]);
for (const [key, value] of Object.entries({ KOMODO_ADDRESS: 'http://komodo.test', KOMODO_API_KEY: 'test', KOMODO_API_SECRET: 'test',
KOMODO_TARGET_SERVER: 'host', RCON_PASSWORD: 'test', DEPLOYMENT_CREATE_CONCURRENCY: '1', DEPLOYMENT_RESTART_CONCURRENCY: '1',
SERVER_PORT_MIN: '28960', SERVER_PORT_MAX: '28970' })) vi.stubEnv(key, value);
const komodo = await import('../../src/lib/server/integrations/komodo.js');
for (const name of ['komodoRestart', 'komodoRuntimeAction', 'komodoCreateDeployment', 'komodoDeploy', 'komodoDestroyDeployment', 'komodoDeleteDeployment'] as const) {
vi.mocked(komodo[name]).mockResolvedValue({ ok: true });
}
vi.mocked(komodo.komodoInspectHealthy).mockResolvedValue({ ok: true, state: 'running' });
vi.mocked(komodo.komodoEnsureTag).mockImplementation(async name => ({ ok: true, id: name }));
vi.mocked(komodo.komodoUpdateTagsOnResource).mockResolvedValue({ ok: true });
const rcon = await import('../../src/lib/server/integrations/rcon.js');
vi.mocked(rcon.rconGetPlayerCount).mockResolvedValue(0);
vi.mocked(rcon.rconStatusResult).mockResolvedValue(status());
vi.mocked(rcon.rconPlutoniumVersion).mockResolvedValue({ revision: 5354, error: null });
});
afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); vi.unstubAllEnvs(); vi.unstubAllGlobals(); });
async function register() {
const { db } = await import('../../src/lib/server/db/index.js');
db().prepare(`INSERT INTO servers (name,komodo_type,komodo_name,game,map,game_mode,map_slug,game_port,host_dir)
VALUES (?, 'deployment', ?, 't5','zombie_theater','zombie','theater',28960,?)`).run(server, server, join(fixture.t5, server));
const { saveServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
saveServerKey(server, 'saved-key');
}
async function detail(id: string) {
const { getDeployment } = await import('../../src/lib/server/provisioning/history.js');
return getDeployment(id)!;
}
async function telemetry(players: number, map = 'zombie_theater') {
const { db } = await import('../../src/lib/server/db/index.js');
const receivedAt = new Date().toISOString();
db().prepare(`INSERT INTO server_telemetry (server,round,players,received_at,payload) VALUES (?,1,?,?,?)
ON CONFLICT(server) DO UPDATE SET players=excluded.players,received_at=excluded.received_at,payload=excluded.payload`)
.run(server, players, receivedAt, JSON.stringify({ ...gameEvent('heartbeat', 1, players, server, map), receivedAt }));
return receivedAt;
}
async function launch(game: 't5' | 't6' = 't5') {
const { queueCreateServer, processCreations } = await import('../../src/lib/server/provisioning/instances.js');
const { rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js');
vi.mocked(rconStatusResult).mockResolvedValue(status(game === 't5' ? 'zombie_theater' : 'zm_transit'));
const job = queueCreateServer({ game, gameMode: 'zombie', mapSlug: game === 't5' ? 'theater' : 'town', plutoKey: 'saved-key' });
if (!job.ok) throw new Error(job.message);
await processCreations();
vi.setSystemTime(Date.now() + 15_000);
return { ...job, processCreations };
}
describe('Plutonium provisioning verification', () => {
it.each(['t5', 't6'] as const)('verifies production and persists the running %s revision before completing', async game => {
const job = await launch(game);
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
const { runningVersions } = await import('../../src/lib/server/integrations/plutonium.js');
const { PLUTONIUM_MANIFEST_URL } = await import('../../src/lib/plutonium.js');
expect(rconPlutoniumVersion).not.toHaveBeenCalled();
await job.processCreations();
const completed = await detail(job.deploymentId);
expect(completed.deployment.state).toBe('done');
expect(completed.progress).toMatchObject({ phase: 'verify-plutonium', blocker: null });
expect(completed.progress?.checks).toEqual(expect.arrayContaining([
expect.objectContaining({ key: 'plutonium-release', source: 'manifest', result: 'pass', actual: 5354 }),
expect.objectContaining({ key: 'plutonium-version', source: 'rcon', result: 'pass', expected: 5354, actual: 5354 })
]));
expect(runningVersions().find(version => version.server === job.name)).toMatchObject({ revision: 5354, update_status: 'current' });
expect(fetch).toHaveBeenCalledWith(PLUTONIUM_MANIFEST_URL, expect.objectContaining({ redirect: 'error' }));
expect(rconPlutoniumVersion).toHaveBeenCalledTimes(1);
expect(rconPlutoniumVersion).toHaveBeenCalledWith(job.name);
const { isProvisioningDetail, provisioningGroups } = await import('../../src/lib/ui/provisioning-presentation.js');
expect(isProvisioningDetail(completed, job.deploymentId)).toBe(true);
expect(provisioningGroups(completed).find(group => group.key === 'verify')?.state).toBe('done');
});
it('retries unknown runtime evidence while retaining startup capacity and never replays creation', async () => {
const job = await launch();
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
const { komodoCreateDeployment, komodoDeploy } = await import('../../src/lib/server/integrations/komodo.js');
const { queueCreateServer } = await import('../../src/lib/server/provisioning/instances.js');
const next = queueCreateServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'second-key' });
if (!next.ok) throw new Error(next.message);
// A pre-start observation is not proof of the newly started process.
const { db } = await import('../../src/lib/server/db/index.js');
db().prepare('INSERT INTO plutonium_versions (server, revision, checked_at) VALUES (?, 5354, ?)')
.run(job.name, '2026-09-28T11:59:00.000Z');
vi.mocked(rconPlutoniumVersion).mockResolvedValueOnce({ revision: null, error: 'Version reply timed out.' });
await job.processCreations();
expect((await detail(job.deploymentId)).deployment.state).toBe('awaiting-health');
expect((await detail(job.deploymentId)).progress?.blocker).toMatchObject({ code: 'plutonium-version', automatic: true });
expect((await detail(next.deploymentId)).deployment.state).toBe('queued');
expect(db().prepare('SELECT revision FROM plutonium_versions WHERE server = ?').get(job.name)).toEqual({ revision: null });
await job.processCreations();
expect((await detail(job.deploymentId)).deployment.state).toBe('done');
expect((await detail(next.deploymentId)).deployment.state).toBe('awaiting-health');
expect(komodoCreateDeployment).toHaveBeenCalledTimes(2);
expect(komodoDeploy).toHaveBeenCalledTimes(2);
});
it('requires a successful manifest check even with a previously cached revision, then resumes without recreating', async () => {
const job = await launch();
const { writeSetting } = await import('../../src/lib/server/maintenance/settings.js');
writeSetting('plutonium_release', { revision: 5300, checkedAt: new Date().toISOString(), error: null });
vi.mocked(fetch).mockRejectedValueOnce(new Error('Manifest offline'));
await job.processCreations();
expect((await detail(job.deploymentId)).deployment.state).toBe('awaiting-health');
expect((await detail(job.deploymentId)).progress?.checks).toContainEqual(expect.objectContaining({
key: 'plutonium-release', result: 'unknown', actual: null, message: 'Manifest offline'
}));
await job.processCreations();
expect((await detail(job.deploymentId)).deployment.state).toBe('done');
const { komodoDeploy } = await import('../../src/lib/server/integrations/komodo.js');
expect(komodoDeploy).toHaveBeenCalledTimes(1);
});
it('reports a confirmed outdated runtime without automatically restarting or recreating it', async () => {
const job = await launch();
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
vi.mocked(rconPlutoniumVersion).mockResolvedValue({ revision: 5300, error: null });
await job.processCreations();
const failed = await detail(job.deploymentId);
expect(failed.deployment).toMatchObject({ state: 'failed', error: expect.stringContaining('running r5300, production is r5354') });
expect(failed.progress?.checks).toContainEqual(expect.objectContaining({ key: 'plutonium-version', result: 'fail', expected: 5354, actual: 5300 }));
const { runningVersions } = await import('../../src/lib/server/integrations/plutonium.js');
expect(runningVersions().find(version => version.server === job.name)).toMatchObject({ revision: 5300, update_status: 'outdated' });
await job.processCreations();
const { komodoCreateDeployment, komodoDeploy, komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
expect(komodoDeploy).toHaveBeenCalledTimes(1);
expect(komodoRestart).not.toHaveBeenCalled();
});
it('accepts a runtime ahead of the published release without requesting a downgrade', async () => {
const job = await launch();
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
vi.mocked(rconPlutoniumVersion).mockResolvedValue({ revision: 5355, error: null });
await job.processCreations();
expect((await detail(job.deploymentId)).deployment.state).toBe('done');
const { runningVersions } = await import('../../src/lib/server/integrations/plutonium.js');
expect(runningVersions().find(version => version.server === job.name)?.update_status).toBe('ahead');
});
it('discards a late version response when the instance was removed during the probe', async () => {
const job = await launch();
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
const { db } = await import('../../src/lib/server/db/index.js');
vi.mocked(rconPlutoniumVersion).mockImplementationOnce(async () => {
db().prepare('DELETE FROM servers WHERE name = ?').run(job.name);
return { revision: 5354, error: null };
});
await job.processCreations();
const removed = await detail(job.deploymentId);
expect(removed.deployment.server_deleted_at).toBeTruthy();
expect(removed.deployment.state).not.toBe('done');
expect(removed.progress?.checkpoints.some(check => check.key === 'verify-plutonium')).toBe(false);
expect(db().prepare('SELECT * FROM plutonium_versions WHERE server = ?').get(job.name)).toBeUndefined();
});
});
describe('provisioning decision evidence', () => {
it('distinguishes queued recipes, attempted files, transient map checks, and telemetry completion without replaying startup', async () => {
const { queueCreateServer, processCreations } = await import('../../src/lib/server/provisioning/instances.js');
const { komodoCreateDeployment, komodoDeploy, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
const { rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js');
const job = queueCreateServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'saved-key' });
if (!job.ok) throw new Error(job.message);
expect((await detail(job.deploymentId)).progress).toMatchObject({ phase: 'startup-queue', recipeStatus: 'queued', observedAt: null });
const { createScript, getLatestVersion } = await import('../../src/lib/server/fleet/scripts.js');
const { db } = await import('../../src/lib/server/db/index.js');
const { script } = createScript({ name: 'New promotion', targetRelPath: 'promoted.gsc', scopeType: 'game', scopeValue: 't5', content: Buffer.from('promoted while queued') });
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(getLatestVersion(script.id)!.id, script.id);
await processCreations();
expect((await detail(job.deploymentId)).progress?.recipeStatus).toBe('attempted');
expect((await detail(job.deploymentId)).recipe?.files.some(file => file.path === 'promoted.gsc')).toBe(true);
vi.mocked(rconStatusResult).mockResolvedValueOnce(status('zombie_moon'));
vi.setSystemTime(Date.now() + 1000);
await processCreations();
expect((await detail(job.deploymentId)).progress).toMatchObject({ phase: 'verify-map', blocker: { code: 'map-mismatch' },
checks: [{ key: 'container', result: 'pass' }, { key: 'game', result: 'pass' }, { key: 'map', result: 'waiting', expected: 'zombie_theater', actual: 'zombie_moon' }] });
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: false, reason: 'container recovering' });
await processCreations();
expect((await detail(job.deploymentId)).progress?.checks).toEqual([expect.objectContaining({ key: 'container', result: 'waiting' })]);
vi.setSystemTime(Date.now() + 1000);
const receivedAt = await telemetry(0);
await processCreations();
const completed = await detail(job.deploymentId);
expect(completed.deployment.state).toBe('done');
expect(completed.progress?.checks).toContainEqual(expect.objectContaining({ key: 'game', source: 'telemetry', result: 'pass', receivedAt }));
expect(completed.progress?.checkpoints).toContainEqual(expect.objectContaining({ key: 'verify-map' }));
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
expect(komodoDeploy).toHaveBeenCalledTimes(1);
});
it('keeps an uncertain failed request immutable while observing and releasing its exact recovery slot', async () => {
await register();
const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js');
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoRestart).mockResolvedValueOnce({ ok: false, kind: 'unreachable', message: 'response lost' });
const job = requestLifecycle(server, 'restart');
if (!job.ok) throw new Error(job.error);
await processProvisioning();
const failed = await detail(job.deploymentId);
expect(failed.progress).toMatchObject({ command: { state: 'uncertain' }, recovery: { state: 'held', operationId: job.deploymentId } });
const failureEvents = failed.events.filter(event => event.kind === 'outcome');
vi.setSystemTime(Date.now() + 15_000);
await telemetry(0);
await processProvisioning();
const recovered = await detail(job.deploymentId);
expect(recovered.deployment).toMatchObject({ state: 'failed', finished_at: failed.deployment.finished_at });
expect(recovered.progress).toMatchObject({ command: { state: 'uncertain' }, recovery: { state: 'released', operationId: job.deploymentId } });
expect(recovered.events.filter(event => event.kind === 'outcome')).toEqual(failureEvents);
expect(komodoRestart).toHaveBeenCalledTimes(1);
});
it('records occupied and unknown evidence without treating either as zero or resetting eligibility timing', async () => {
await register();
const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js');
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
const job = requestLifecycle(server, 'restart');
if (!job.ok) throw new Error(job.error);
vi.mocked(rconGetPlayerCount).mockResolvedValueOnce(null);
await processProvisioning();
const unknown = await detail(job.deploymentId);
expect(unknown.progress?.checks).toEqual([expect.objectContaining({ key: 'occupancy', actual: null, source: null, result: 'unknown' })]);
vi.setSystemTime(Date.now() + 1000);
await telemetry(3);
await processProvisioning();
expect((await detail(job.deploymentId)).progress).toMatchObject({ phaseStartedAt: unknown.progress?.phaseStartedAt,
blocker: { code: 'players-present' }, checks: [{ key: 'occupancy', actual: 3, source: 'telemetry' }] });
vi.setSystemTime(Date.now() + 1000);
await telemetry(2);
await processProvisioning();
expect((await detail(job.deploymentId)).progress?.phaseStartedAt).toBe(unknown.progress?.phaseStartedAt);
expect(komodoRestart).not.toHaveBeenCalled();
});
it('preserves failed cleanup evidence through same-ID resume without another destructive command', async () => {
await register();
const { db } = await import('../../src/lib/server/db/index.js');
const { destroyServer, processDestructions } = await import('../../src/lib/server/provisioning/instances.js');
const { resumeCleanup } = await import('../../src/lib/server/provisioning/history.js');
const { komodoInspectHealthy, komodoDestroyDeployment, komodoDeleteDeployment } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'exited', reason: 'stopped' });
db().exec("CREATE TRIGGER fail_cleanup BEFORE DELETE ON servers BEGIN SELECT RAISE(ABORT, 'cleanup interrupted'); END");
const job = destroyServer(server);
if (!job.ok || !job.deploymentId) throw new Error('Could not queue removal');
await processDestructions();
const failed = await detail(job.deploymentId);
expect(failed.deployment.state).toBe('failed');
expect(failed.progress?.checkpoints).toContainEqual(expect.objectContaining({ key: 'delete-deployment' }));
expect(failed.progress?.checkpoints.some(point => point.key === 'cleanup')).toBe(false);
const failure = failed.events.find(event => event.kind === 'outcome')!;
db().exec('DROP TRIGGER fail_cleanup');
vi.setSystemTime(Date.now() + 1000);
expect(resumeCleanup(job.deploymentId)).toBe(true);
await processDestructions();
const completed = await detail(job.deploymentId);
expect(completed.deployment).toMatchObject({ id: job.deploymentId, state: 'done', server_exists: 0 });
expect(completed.events).toContainEqual(failure);
expect(completed.events.some(event => event.kind === 'resumed')).toBe(true);
expect(completed.progress?.checkpoints).toContainEqual(expect.objectContaining({ key: 'cleanup' }));
expect(komodoDestroyDeployment).toHaveBeenCalledTimes(1);
expect(komodoDeleteDeployment).toHaveBeenCalledTimes(1);
});
it('links the actual removal dependency while later eligible creation bypasses it', async () => {
const { db } = await import('../../src/lib/server/db/index.js');
const { beginDeployment } = await import('../../src/lib/server/provisioning/history.js');
const { queueCreateServer, processCreations } = await import('../../src/lib/server/provisioning/instances.js');
const input = { gameMode: 'zombie' as const, mapSlug: 'theater', plutoKey: 'saved-key' };
const first = queueCreateServer(input), second = queueCreateServer(input);
if (!first.ok || !second.ok) throw new Error('Could not queue creation');
const removal = beginDeployment('old-instance', 'destroy');
db().prepare('INSERT INTO server_creation_dependencies VALUES (?, ?)').run(first.deploymentId, removal);
await processCreations();
expect((await detail(first.deploymentId)).progress?.blocker).toMatchObject({ code: 'removal-dependency', href: `/provisioning/${removal}` });
expect((await detail(first.deploymentId)).deployment.state).toBe('queued');
expect((await detail(second.deploymentId)).deployment.state).toBe('awaiting-health');
});
it.each(['create', 'delete'] as const)('records uncertainty when the %s reply throws without inventing a successful checkpoint', async command => {
const komodo = await import('../../src/lib/server/integrations/komodo.js');
const { queueCreateServer, processCreations, destroyServer, processDestructions } = await import('../../src/lib/server/provisioning/instances.js');
let id: string;
if (command === 'create') {
vi.mocked(komodo.komodoCreateDeployment).mockRejectedValueOnce(new Error('reply lost'));
const job = queueCreateServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'saved-key' });
if (!job.ok) throw new Error(job.message);
id = job.deploymentId;
await processCreations();
expect(komodo.komodoCreateDeployment).toHaveBeenCalledTimes(1);
} else {
await register();
vi.mocked(komodo.komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'exited', reason: 'stopped' });
vi.mocked(komodo.komodoDeleteDeployment).mockRejectedValueOnce(new Error('reply lost'));
const job = destroyServer(server);
if (!job.ok || !job.deploymentId) throw new Error('Could not queue removal');
id = job.deploymentId;
await processDestructions();
expect(komodo.komodoDeleteDeployment).toHaveBeenCalledTimes(1);
}
const failed = await detail(id);
expect(failed.deployment.state).toBe('failed');
expect(failed.progress?.command).toMatchObject({ state: 'uncertain', message: expect.stringContaining('reply lost') });
expect(failed.progress?.checkpoints.some(point => point.key === (command === 'create' ? 'create-deployment' : 'delete-deployment'))).toBe(false);
});
it.each(['destroy', 'rebuild'] as const)('retains the resumable %s checkpoint when a beta arrives after confirmed remote deletion', async action => {
await register();
const { db } = await import('../../src/lib/server/db/index.js');
const { createScript } = await import('../../src/lib/server/fleet/scripts.js');
const { destroyServer, processDestructions } = await import('../../src/lib/server/provisioning/instances.js');
const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js');
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
const { resumeCleanup, resumeRebuild, REMOVING_SERVER } = await import('../../src/lib/server/provisioning/history.js');
const { komodoInspectHealthy, komodoDestroyDeployment, komodoDeleteDeployment } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'exited', reason: 'stopped' });
vi.mocked(komodoDeleteDeployment).mockImplementationOnce(async () => {
const { script, version } = createScript({ name: 'Late beta', targetRelPath: 'beta.gsc', scopeType: 'game', scopeValue: 't5', content: Buffer.from('beta') });
const beta = db().prepare("INSERT INTO betas (script_id,script_version_id,trigger_kind,state) VALUES (?,?,'manual','running')").run(script.id, version.id).lastInsertRowid;
db().prepare("INSERT INTO beta_servers (beta_id,server,had_original,stage) VALUES (?,?,0,'ready')").run(beta, server);
return { ok: true };
});
const job = action === 'destroy' ? destroyServer(server) : requestLifecycle(server, 'rebuild');
if (!job.ok || !job.deploymentId) throw new Error('Could not queue removal');
await (action === 'destroy' ? processDestructions() : processProvisioning());
const failed = await detail(job.deploymentId);
expect(failed.deployment).toMatchObject({ state: 'failed', stage: REMOVING_SERVER });
expect(failed.progress?.checkpoints).toContainEqual(expect.objectContaining({ key: 'delete-deployment' }));
expect(action === 'destroy' ? failed.canResumeCleanup : failed.canResumeRebuild).toBe(true);
db().prepare("UPDATE betas SET state = 'rolled-back'").run();
expect(action === 'destroy' ? resumeCleanup(job.deploymentId) : resumeRebuild(job.deploymentId)).toBe(true);
await (action === 'destroy' ? processDestructions() : processProvisioning());
expect((await detail(job.deploymentId)).deployment.state).toBe(action === 'destroy' ? 'done' : 'queued');
expect(komodoDestroyDeployment).toHaveBeenCalledTimes(1);
expect(komodoDeleteDeployment).toHaveBeenCalledTimes(1);
});
it('ignores occupancy evidence that returns after the request was cancelled', async () => {
await register();
const { requestLifecycle, cancelProvisioning } = await import('../../src/lib/server/provisioning/requests.js');
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
const job = requestLifecycle(server, 'restart');
if (!job.ok) throw new Error(job.error);
vi.mocked(rconGetPlayerCount).mockImplementationOnce(async () => {
expect(cancelProvisioning(job.deploymentId)).toBe(true);
return 4;
});
await processProvisioning();
const cancelled = await detail(job.deploymentId);
expect(cancelled.deployment.state).toBe('cancelled');
expect(cancelled.progress).toMatchObject({ checks: [], blocker: null });
expect(komodoRestart).not.toHaveBeenCalled();
});
it('does not attach replacement health to a deleted creation after a delayed probe', async () => {
const { db } = await import('../../src/lib/server/db/index.js');
const { queueCreateServer, processCreations } = await import('../../src/lib/server/provisioning/instances.js');
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
const { rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js');
const job = queueCreateServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'saved-key' });
if (!job.ok) throw new Error(job.message);
await processCreations();
vi.mocked(komodoInspectHealthy).mockImplementationOnce(async () => {
db().prepare('DELETE FROM servers WHERE name = ?').run(server);
await register();
return { ok: true, state: 'running' };
});
await processCreations();
const old = await detail(job.deploymentId);
expect(old.deployment.server_deleted_at).toBeTruthy();
expect(old.progress?.checks).toEqual([]);
expect(old.progress?.checkpoints.some(point => point.key.startsWith('verify-'))).toBe(false);
expect(rconStatusResult).not.toHaveBeenCalled();
});
});
@@ -24,9 +24,9 @@ describe('provisioning reads with retained lifecycle history', () => {
const now = Date.parse('2026-10-05T12:00:00Z'); const now = Date.parse('2026-10-05T12:00:00Z');
const iso = (offset: number) => new Date(now + offset).toISOString(); const iso = (offset: number) => new Date(now + offset).toISOString();
handle.transaction(() => { handle.transaction(() => {
const server = handle.prepare('INSERT INTO servers(name, komodo_name) VALUES (?, ?)'); const server = handle.prepare('INSERT INTO servers(name) VALUES (?)');
for (let i = 0; i < 16; i++) server.run(`server-${i}`, `server-${i}`); for (let i = 0; i < 16; i++) server.run(`server-${i}`);
server.run('empty', 'empty'); server.run('empty');
const insert = handle.prepare(`INSERT INTO server_deployments const insert = handle.prepare(`INSERT INTO server_deployments
(id, server, action, state, stage, started_at, finished_at, submitted_at, server_deleted_at, triggered_by) (id, server, action, state, stage, started_at, finished_at, submitted_at, server_deleted_at, triggered_by)
VALUES (?, ?, ?, 'done', 'Complete', ?, ?, ?, ?, 'test')`); VALUES (?, ?, ?, 'done', 'Complete', ?, ?, ?, ?, 'test')`);
@@ -121,7 +121,7 @@ describe('provisioning reads with retained lifecycle history', () => {
legacy.exec(schema.replace(/CREATE TABLE IF NOT EXISTS server_deployments \([\s\S]*?\n\);/, table => table legacy.exec(schema.replace(/CREATE TABLE IF NOT EXISTS server_deployments \([\s\S]*?\n\);/, table => table
.replace("'create', 'start', 'restart', 'rebuild', 'stop', 'destroy'", "'create', 'destroy'") .replace("'create', 'start', 'restart', 'rebuild', 'stop', 'destroy'", "'create', 'destroy'")
.replace(' server_deleted_at TEXT,\n', ''))); .replace(' server_deleted_at TEXT,\n', '')));
legacy.exec(`INSERT INTO servers(name,komodo_name) VALUES('existing','existing'); legacy.exec(`INSERT INTO servers(name) VALUES('existing');
INSERT INTO server_deployments(id,server,action,state,stage,started_at,triggered_by) INSERT INTO server_deployments(id,server,action,state,stage,started_at,triggered_by)
VALUES('old','existing','create','done','Complete','2025-01-01','fixture'); VALUES('old','existing','create','done','Complete','2025-01-01','fixture');
INSERT INTO server_deployment_events(deployment_id,created_at,message) INSERT INTO server_deployment_events(deployment_id,created_at,message)
+6 -7
View File
@@ -72,7 +72,7 @@ describe('provisioning outcome and evidence presentation', () => {
it('shows passed container and telemetry game checks while waiting on the engine map', () => { it('shows passed container and telemetry game checks while waiting on the engine map', () => {
const detail = fixture(); const detail = fixture();
detail.progress = progress({ checks: [ detail.progress = progress({ checks: [
{ key: 'container', label: 'Container readiness', result: 'pass', source: 'komodo', checkedAt: at, receivedAt: null, expected: 'healthy', actual: 'healthy', message: null }, { key: 'container', label: 'Container readiness', result: 'pass', source: 'agent', checkedAt: at, receivedAt: null, expected: 'healthy', actual: 'healthy', message: null },
{ key: 'game', label: 'Game status', result: 'pass', source: 'telemetry', checkedAt: at, receivedAt: at, expected: null, actual: null, message: 'Fresh telemetry accepted' }, { key: 'game', label: 'Game status', result: 'pass', source: 'telemetry', checkedAt: at, receivedAt: at, expected: null, actual: null, message: 'Fresh telemetry accepted' },
{ key: 'map', label: 'Engine map', result: 'waiting', source: 'telemetry', checkedAt: at, receivedAt: at, expected: 'zombie_theater', actual: 'zombie_moon', message: null } { key: 'map', label: 'Engine map', result: 'waiting', source: 'telemetry', checkedAt: at, receivedAt: at, expected: 'zombie_theater', actual: 'zombie_moon', message: null }
] }); ] });
@@ -131,14 +131,13 @@ describe('provisioning outcome and evidence presentation', () => {
expect(provisioningGroups(detail).flatMap(group => group.steps).find(step => step.key === 'verify-game')?.state).toBe('done'); expect(provisioningGroups(detail).flatMap(group => group.steps).find(step => step.key === 'verify-game')?.state).toBe('done');
}); });
it('keeps removal checkpoints separate from replacement creation', () => { it('keeps confirmed agent removal separate from registry cleanup', () => {
const detail = fixture(); detail.deployment.action = 'rebuild'; detail.deployment.state = 'waiting'; const detail = fixture(); detail.deployment.action = 'destroy'; detail.deployment.state = 'waiting';
detail.progress = progress({ phase: 'cleanup', checkpoints: [{ key: 'delete-deployment', label: 'Remote deployment deleted', at, phase: 'delete-deployment' }] }); detail.progress = progress({ phase: 'cleanup', checkpoints: [{ key: 'container-action', label: 'Agent confirmed removal', at, phase: 'container-action' }] });
const groups = provisioningGroups(detail); const groups = provisioningGroups(detail);
expect(groups.map(group => group.key)).toEqual(['request', 'remove', 'create', 'verify']); expect(groups.map(group => group.key)).toEqual(['request', 'command', 'cleanup']);
expect(groups.flatMap(group => group.steps).find(step => step.key === 'delete-deployment')?.state).toBe('done'); expect(groups.flatMap(group => group.steps).find(step => step.key === 'container-action')?.state).toBe('done');
expect(groups.flatMap(group => group.steps).find(step => step.key === 'cleanup')?.state).toBe('waiting'); expect(groups.flatMap(group => group.steps).find(step => step.key === 'cleanup')?.state).toBe('waiting');
expect(groups.flatMap(group => group.steps).find(step => step.key === 'prepare')?.state).toBe('pending');
}); });
}); });
+3 -3
View File
@@ -43,10 +43,10 @@ describe('provisioning evidence persistence', () => {
const { handle, recordProgress, read } = await setup(); const { handle, recordProgress, read } = await setup();
recordProgress('operation', { phase: 'verify-game', command: { state: 'uncertain', message: 'Reply was lost' }, recordProgress('operation', { phase: 'verify-game', command: { state: 'uncertain', message: 'Reply was lost' },
checkpoint: { key: 'container-action', label: 'Submission attempted', phase: 'container-action' }, checkpoint: { key: 'container-action', label: 'Submission attempted', phase: 'container-action' },
checks: [{ key: 'container', label: 'Container', source: 'komodo', result: 'pass' }, checks: [{ key: 'container', label: 'Container', source: 'agent', result: 'pass' },
{ key: 'game', label: 'Game', source: 'telemetry', result: 'pass' }] }); { key: 'game', label: 'Game', source: 'telemetry', result: 'pass' }] });
vi.setSystemTime(Date.now() + 15_000); vi.setSystemTime(Date.now() + 15_000);
recordProgress('operation', { phase: 'verify-container', checks: [{ key: 'container', label: 'Container', source: 'komodo', result: 'unknown' }] }); recordProgress('operation', { phase: 'verify-container', checks: [{ key: 'container', label: 'Container', source: 'agent', result: 'unknown' }] });
expect(read()?.checks).toHaveLength(1); expect(read()?.checks).toHaveLength(1);
expect(read()?.checks[0].result).toBe('unknown'); expect(read()?.checks[0].result).toBe('unknown');
expect(read()?.checkpoints).toHaveLength(1); expect(read()?.checkpoints).toHaveLength(1);
@@ -87,7 +87,7 @@ describe('provisioning evidence persistence', () => {
const { handle, recordProgress, read } = await setup(); const { handle, recordProgress, read } = await setup();
expect(() => handle.transaction(() => { expect(() => handle.transaction(() => {
handle.prepare("UPDATE server_deployments SET state = 'running' WHERE id = 'operation'").run(); handle.prepare("UPDATE server_deployments SET state = 'running' WHERE id = 'operation'").run();
recordProgress('operation', { phase: 'cleanup', checkpoint: { key: 'delete-deployment', label: 'Remote deletion confirmed' } }); recordProgress('operation', { phase: 'cleanup', checkpoint: { key: 'container-action', label: 'Remote deletion confirmed' } });
throw new Error('Interrupted transaction'); throw new Error('Interrupted transaction');
})()).toThrow('Interrupted transaction'); })()).toThrow('Interrupted transaction');
expect(read()).toBeNull(); expect(read()).toBeNull();

Some files were not shown because too many files have changed in this diff Show More