Complete provisioning job detail pages and lifecycle evidence

This commit is contained in:
2026-09-28 14:17:19 -04:00
parent 5b65520bbc
commit 3fb59cd598
41 changed files with 2866 additions and 346 deletions
+10 -1
View File
@@ -433,7 +433,16 @@ distinguished.
Provisioning (`/provisioning`, `/api/provisioning`) owns Create, Start, Restart, Rebuild, Stop, and Destroy: request persistence, eligibility and occupancy checks, execution, recovery, and history. Campaigns, betas, and scheduled maintenance request lifecycle work through Provisioning and link to the resulting operation. They retain their file, test-evidence, and version-verification workflows. Pending automated restarts appear in Provisioning before execution. History survives server removal. New creation requests return HTTP 202 with an operation ID and open the provisioning detail page. Creation is complete only when Komodo reports healthy and RCON or fresh telemetry verifies game status with an engine map that exactly matches the selected map. Queued requests reserve their server name and port; queued creation and pending health checks survive application restarts. The detail page records startup progress and explains any health or map mismatch that is still blocking completion. Provisioning (`/provisioning`, `/api/provisioning`) owns Create, Start, Restart, Rebuild, Stop, and Destroy: request persistence, eligibility and occupancy checks, execution, recovery, and history. Campaigns, betas, and scheduled maintenance request lifecycle work through Provisioning and link to the resulting operation. They retain their file, test-evidence, and version-verification workflows. Pending automated restarts appear in Provisioning before execution. History survives server removal. New creation requests return HTTP 202 with an operation ID and open the provisioning detail page. Creation is complete only when Komodo reports healthy and RCON or fresh telemetry verifies game status with an engine map that exactly matches the selected map. Queued requests reserve their server name and port; queued creation and pending health checks survive application restarts. The detail page records startup progress and explains any health or map mismatch that is still blocking completion.
Each server deployment opens a detail page with its flow and timestamped activity. Destroy uses an in-app confirmation modal and immediately returns a queued deployment (`DELETE /api/servers/:name` returns HTTP 202 with `deploymentId`). The worker waits for a fresh zero-player RCON or telemetry report, or a verified stopped container, before teardown. Occupied reports from either source block teardown; servers without current verification remain queued and are checked each worker tick. Waiting requests survive application restarts, while interrupted teardown is marked for inspection. Completed destruction stays on the provisioning detail page. Each server deployment opens a detail page with its outcome, blocker, permitted
recovery actions, grouped execution phases, and timestamped verification evidence.
New jobs record command certainty and the source of game-health checks; older
records retain their known outcomes without invented evidence. A failed restart's
ongoing recovery and held capacity appear separately from its recorded failure.
Recent activity is visible immediately, with searchable history loaded in pages.
Launch files distinguish the setup captured while queued from the files selected
when creation began. The page retains its last snapshot if a refresh fails.
Destroy uses an in-app confirmation modal and immediately returns a queued deployment (`DELETE /api/servers/:name` returns HTTP 202 with `deploymentId`). The worker waits for a fresh zero-player RCON or telemetry report, or a verified stopped container, before teardown. Occupied reports from either source block teardown; servers without current verification remain queued and are checked each worker tick. Waiting requests survive application restarts, while interrupted teardown is marked for inspection. Completed destruction stays on the provisioning detail page.
Use **Provisioning → Manage server**, or a server’s **Server actions**, to start, Use **Provisioning → Manage server**, or a server’s **Server actions**, to start,
+44
View File
@@ -0,0 +1,44 @@
# Provisioning detail implementation
Implemented September 28, 2026, following the [approved review plan](provisioning-improvement-plan.md).
## Delivered behavior
The job page now leads with identity, recorded outcome, blocker, and permitted recovery action. Grouped execution phases and timestamped evidence precede supporting metadata and launch files. Mobile follows the same reading order. Operation IDs and links are copyable, and the return link preserves Provisioning filters or leads back to related work.
The worker records versioned progress snapshots and semantic events at existing decision points. Container, game, map, and occupancy checks identify their source and observation time. Durable milestones preserve remote deletion, local cleanup, command certainty, and operation-owned recovery capacity. A failed restart can retain its failed outcome while later recovery evidence records capacity release. Legacy records retain their original stage and outcome without invented checkpoints.
Cleanup resumes honor backend capabilities even for deleted identities. Confirmed remote deletion and the existing cleanup-resume stage are saved together before local cleanup, preserving recovery if a later guard or local operation fails. Late observations are rejected after cancellation, deletion, or replacement. Existing occupancy rules, scheduling, command counts, and observation-only resumes are covered by regression tests.
The page uses server-loaded detail and abortable, visit-scoped refreshes. Action feedback survives background refresh failures; outdated requests cannot replace another job's detail. Terminal records continue visibility-aware polling because recovery, deletion, and capabilities can still change.
Five recent events are visible. Full history loads in 25-entry pages with literal search, ordering, and a fixed upper event ID. Recent activity stays still while full history is open; new events appear through an explicit Jump to latest control. The legacy detail API still returns full events by default. The new page opts into `?events=recent`.
Queued recipe captures and files selected for an attempt have separate labels and timestamps. Neither implies successful installation. Related work names resolve when available, while missing source IDs remain visible without broken links.
## Validation
- Full suite: **898 tests passed across 72 files**.
- Svelte/TypeScript: `npm run check` passed with **0 errors and 0 warnings**.
- Production: `npm run build` passed with the Node adapter.
- Database tests cover additive migration, historical identity and event preservation, transaction rollback, stable phase timing, semantic outcomes across same-ID resumes, and full versus bounded API compatibility.
- Lifecycle tests cover ambiguous and thrown remote replies, telemetry and map evidence, cancelled/replaced identity races, exact recovery ownership, and destroy/rebuild cleanup after confirmed deletion without repeating remote commands.
- Refresh/presentation tests cover navigation and action races, persistent acknowledged action results, failed reconciliation, cancellation, deleted cleanup, legacy evidence, and malformed responses.
- Firefox checks exercised history pagination, literal search, stable new-event handling, failed-refresh retention and recovery, and layouts at 1440, 390, and 320 pixels without page-wide overflow. Enter and Space operate phase disclosures. A deleted-identity cleanup resume completed through the real local API; retained restart recovery, cancellation, and legacy evidence also rendered correctly.
- Reflow also passed at 720 CSS pixels with pixel ratio 2, and long unbroken identifiers fit at 320px. This simulates the available layout width of 200% zoom on a 1440px display; native browser zoom was not measured. A separate CSS-zoom probe kept the detail content in bounds but exposed overflow in the existing global navigation.
Browser fixtures use a temporary database and disabled external integrations. They exercise the real page and local detail/history APIs; lifecycle tests stub remote systems. This is not a production fleet smoke test or a formal accessibility conformance audit. The build reports the repository's existing `csrf.checkOrigin` deprecation notice.
## Measurements and screenshots
For a synthetic 5,001-event operation, full detail serialized to **1,884,119 bytes** and bounded detail to **3,011 bytes**. Thirty local bounded reads averaged **2.93 ms**. These measurements establish payload reduction on the local fixture, not a production latency guarantee; event totals and search still query retained history.
In the 1440px create/map-wait fixture, the execution heading moved from approximately **1,073px** in the original composition to **675px**. The page keeps the outcome above execution and the recipe in a supporting column. Phone layouts move context below execution and activity.
Updated screenshots: [desktop create/map wait](provisioning-review-assets/implemented-create-desktop.png), [mobile create/map wait](provisioning-review-assets/implemented-create-mobile.png), [deleted identity with permitted cleanup](provisioning-review-assets/implemented-deleted-cleanup.png), and [failed restart with retained recovery](provisioning-review-assets/implemented-restart-recovery.png). The original review captures remain linked from the approved plan.
## Compatibility and rollout
The schema additions are nullable, and old stage text and default API event responses remain supported. Progress is written as new work is evaluated; historical operations without structured evidence use the legacy presentation. Detail reads perform no remote probes.
The change is implemented in the workspace. Normal deployment should verify one newly instrumented job and one legacy record against the configured integrations. No production deployment was performed during this work.
+230
View File
@@ -0,0 +1,230 @@
# Provisioning job detail improvement plan
Review date: September 28, 2026. Source baseline: `5b65520`.
The recommendation is to make `/provisioning/:id` a complete operational record: identify the job, explain its current outcome and blocker, expose the evidence behind that outcome, and show the next permitted action. Progress and recovery should precede historical launch files.
Approved and implemented on September 28, 2026. The original review and agreed scope remain below; see the [implementation and validation record](provisioning-implementation.md) for delivered behavior, checks, and updated screenshots.
## Review method and participants
The requested panel covered three UI/UX roles, two CI/CD systems engineering roles, and one software engineering role. The platform allowed only three child-agent threads and rejected additional spawns even after those reviews completed. The three threads therefore performed two separate rounds, first UX and then engineering. This provides six role-based reviews, rather than six independent agents. Each round included direct peer discussion; engineering reviewed and challenged the UX proposals.
| Review role | Proposal and contribution |
| --- | --- |
| UI/UX: information architecture | Prioritize job identity, outcome, blocker, safe action, progress, activity, and broader work context. Separate historical completion from current server health. |
| UI/UX: interaction and accessibility | Use state-specific recovery actions, stable refresh behavior, persistent action feedback, keyboard-accessible progress, and usable history while updates arrive. |
| UI/UX: visual design | Replace the overloaded summary and wrapping step chips with a compact outcome card, grouped vertical progress, and supporting metadata. Reuse the existing design system. |
| CI/CD systems: lifecycle and queues | Preserve destructive checkpoints, occupancy gates, scheduling, command certainty, identity guards, and recovery-slot semantics. Make queue claims match actual admission behavior. |
| CI/CD systems: observability and reliability | Record the source and age of verification evidence, distinguish page freshness from worker checks, preserve recovery evidence, and validate compatibility and failure scenarios. |
| Software engineering | Define shared types and component boundaries, reuse server-loaded detail, fix route/poll/action races, and stage additive implementation with regression coverage. |
The review covered the page, shared history component, lifecycle APIs, worker, queues, persistence, related collection/maintenance/campaign pages, and relevant tests. Root also inspected the current source in Firefox using an isolated temporary database, disabled integration credentials, and synthetic detail responses. Desktop captures used 1440px width; mobile used 390px. These are composed UI scenarios, not executions of real provisioning workflows or measurements of production use.
Review assets: [create waiting for a map](provisioning-review-assets/create-map-wait-desktop.png), [mobile rebuild composition](provisioning-review-assets/rebuild-mobile.png), and [deleted identity with resumable cleanup](provisioning-review-assets/deleted-cleanup-desktop.png).
## What feels incomplete today
The page already has useful recovery buttons, server diagnostics links, a recorded launch recipe, source backlinks, elapsed time, progress steps, and searchable activity. Preserve those capabilities. The gaps concern their priority and the reliability of the story they tell.
| Finding | Effect on the operator | Evidence |
| --- | --- | --- |
| One summary mixes status, actions, errors, warnings, diagnostics, map, timing, and provenance. The recipe appears before progress. | Monitoring and recovery compete with secondary information. | [Detail page](../src/routes/provisioning/[id]/+page.svelte), summary and recipe markup. In the synthetic desktop create case, the flow began about 1,073px down. |
| Rebuild renders 12 equally weighted wrapping step chips. | Removal, cleanup, and fresh creation do not read as distinct phases. | [Detail page](../src/routes/provisioning/[id]/+page.svelte), `steps`; [OperationHistory](../src/lib/ui/components/OperationHistory.svelte). |
| Progress and step duration depend on matching human-readable stage/event text. | New wording can move the highlighted step backward; changes in player count can appear to restart a step timer. | Detail page `currentStep` and `stageStarted`; [creation verification](../src/lib/server/provisioning/instances.ts) now emits “RCON or fresh telemetry” stages. |
| Runtime cancellation is handled after runtime-specific explanation logic. | A cancelled start/restart/stop can still claim automatic verification is underway. | Detail page `explanation`. |
| The deleted-server flag suppresses every cleanup button and the active-state presentation. | A permitted local cleanup can become inaccessible or appear inactive after resuming. | Detail page recovery controls and `active` versus [history.ts](../src/lib/server/provisioning/history.ts), `canResumeCleanup`. Missing button reproduced with a synthetic permitted response. |
| Completion copy asserts RCON health in the present tense. | Telemetry-based verification is misreported, and historical success can look like current health. | Detail page explanations; [status-evidence.ts](../src/lib/server/fleet/status-evidence.ts), `probeServerStatus`. |
| One refresh guard and one error string handle reads, mutations, and navigation. | A post-action reload can be dropped, an old request can affect another job, and polling can erase an action error. | Detail page `load`, `resume`, `cancelRemoval`; source-inferred race cases. |
| A failed restart can still hold a recovery slot. | A simple failed badge does not explain continued observation or effects on other jobs. | [restarts.ts](../src/lib/server/provisioning/restarts.ts), `refreshRestartRollout`. |
| Activity is collapsed, operation ID is absent from the page body, and related-work links are scattered. | Investigation and handoff require avoidable searching and navigation. | Detail page and shared history component. |
## Proposed page
Use one reading order across all six actions. At wide widths, metadata and launch inputs can occupy a supporting column. On phones they follow progress and activity in the DOM. Avoid a sticky sidebar.
```text
Provisioning / Restart server
Origins · Lobby 1 t6_zombie_origins_1
Operation ID [Copy] [Copy link] [View server]
[Waiting] Waiting for players to leave
Last evaluated: 3 players · source and observation time
Continues automatically after an eligible empty/stopped check.
Relevant permitted action or dependency link
Total elapsed · Time in phase · Page updated / Refresh
EXECUTION AND EVIDENCE OPERATION CONTEXT
Eligibility Requested by / requested at
Container action Submission attempted / finished
Readiness verification Related campaign/beta/maintenance
Container Collection launch
Game status
Selected map, when required Recorded launch files
[Launch another server]
RECENT ACTIVITY
Five latest entries + relevant failure
[Full searchable history]
```
The example evidence requires the new contract below; do not populate it from guessed values.
### Identity, outcome, and next action
- Show the action, friendly map/lobby title, technical server name, and copyable operation ID/link. Use existing identity and copy components.
- Lead with the specific blocker or outcome, followed by what happens next. Place the recorded error and permitted recovery control together. Give warnings a labeled explanation rather than color alone.
- Treat deletion as historical identity context. It must not replace the outcome, hide an actionable cleanup, or link an old operation to a replacement server with the same name.
- Determine active progress from the operation and recovery state, independently of whether a live server link is available.
- Distinguish a recorded outcome from ongoing recovery. A failed request can show “Recovery is still being checked; restart capacity remains reserved” when the backend records that condition.
- Use past-tense completion language with its timestamp: “Creation completed. Container readiness, game status, and the selected engine map were verified at …”. Name RCON or telemetry only when recorded. Current health remains on the server page.
- Preserve an explicit return link to the originating Provisioning filters or parent work, with a safe local fallback. Group all requesting campaigns, betas, maintenance records, and collection launches under Related work. Keep retained source IDs visible when their originating record no longer exists, without a broken link.
- Finishing a restart does not mean its campaign, beta, or maintenance verification has finished. Link back to that work with accurate wording.
### Progress and evidence
Use a vertical checklist with grouped phases, icons, explicit text states, and the current/failed phase expanded. Keep completed phases available for inspection. Avoid percentage complete or predicted finish times.
| Action | Phase sequence and completion evidence |
| --- | --- |
| Create | Request/startup queue → prepare files and create deployment → start container → container readiness → game status → matching engine map. |
| Start | Eligibility and restart capacity → start existing container → container and game verification. Do not imply files were redeployed. |
| Restart | Eligibility, occupancy, and capacity → restart submission → container and game verification. Show any continuing recovery separately from request failure. |
| Rebuild | Eligibility/capacity → remove old container and deployment → local files/registry cleanup → queue fresh instance → create/start → readiness and map verification. Preserve the reserved identity/port and show when the old instance is gone. |
| Stop | Eligibility and occupancy → stop submission → explicit stopped-container confirmation. Explain that files and registration remain. |
| Destroy | Eligibility and occupancy → container removal → deployment deletion → local files/registry cleanup. Retain the historical record after removal. |
Rebuild grouping must retain each irreversible checkpoint. “Remote deployment deleted” and “Local cleanup complete” are different facts; fresh creation must never look complete while old files remain.
Progress supports waiting, current, completed, failed, cancelled/not run, skipped when actually recorded, and unknown. Do not mark earlier steps complete simply because a string matched a later label. Legacy records retain their raw stage and known outcome, with unrecorded milestones labeled accordingly.
The blocker panel shows the worker's actual reason: occupied/unknown players, missing configuration, requester hold, conflicting work, startup or restart capacity, removal/port dependency, container readiness, game evidence, or wrong map. Include the blocking operation/source link when known. Keep “Requested by” separate from “Waiting on.”
Show relevant capacity limits, holders, and this operation's reservation from the same backend rules. If a submission rank is shown, label it as request order: blocked earlier jobs can be bypassed, and source priorities affect admission. Do not say “You are next.”
Evidence rows show result, evaluation time, source, and expected/observed values where relevant. Engine-map verification does not independently verify a T6 variant/start location. Unknown players are never displayed as zero.
Separate durable milestones from transient readiness checks. A passed container/game/map observation remains evidence of what passed at its timestamp; a later evaluation may make the current check failed or unknown. Do not leave stale green checks implying current readiness.
### Actions and terminal states
| Backend condition | Page behavior |
| --- | --- |
| `canCancelRemoval` | Offer **Cancel queued removal**, explaining that it is available only before removal begins. Preserve the existing atomic worker/cancellation race protection. |
| `canResumeVerification` | Offer **Resume health verification** and explain that it observes the existing instance without creating or restarting it. |
| `canResumeCleanup` | Offer **Resume cleanup**, including eligible deleted identities. Explain that Komodo deletion already succeeded. |
| `canResumeRebuild` | Offer **Resume rebuild cleanup** and explain that local cleanup precedes creation in the reserved slot. |
| Failed/interrupted without a resume capability | Lead with the recorded failure and relevant inspection destination. An uncertain remote response must not produce a generic retry command. |
| Cancelled | State that the request is cancelled; distinguish recorded prior work from remaining work that did not run. Do not claim continued execution. |
| Warning | Explain the specific remaining issue. Do not assume every warning concerns cleanup. |
| Complete | Show the verified historical outcome and appropriate View server / Back to related work / Back to history link. |
The API revalidates every action. A 409 triggers reconciliation of current capabilities and explains the state change; it never automatically repeats a mutation. Disable duplicate submissions while an action is pending, keep its accessible name, and preserve its result independently of background refresh. If an acknowledged successful action is followed by a failed refresh, retain its success acknowledgment alongside stale-data feedback; a refresh failure does not turn the action into a failure.
“Launch another server” remains secondary beside the recipe, with its existing explanation that it creates a separate server using current promoted files. It is not a repair or replay of the recorded launch. Create and rebuild already save a recipe while queued, then recapture promoted files when creation starts. Label these **Setup captured when queued** versus **Files selected for this attempt**, with capture times and an explicit recipe status. The queued capture is not a version pin, and selected files do not prove installation succeeded. Legacy records without that distinction retain neutral recorded-recipe wording. Where no recipe exists, explain the absence.
### Activity, refresh, and accessibility
- Keep five recent events visible and show a targeted failure excerpt when needed. Retain full-history search, ordering, and Show more. Do not automatically open a large history when a failure arrives.
- Load full history on demand in bounded pages rather than retransmitting the entire retained history every five seconds. Preserve the legacy API's default response through an opt-in page query.
- Preserve search, disclosure, focus, and reading position across updates. While inspecting older history, show a new-event count and **Jump to latest** instead of moving the content underneath the reader.
- Separate initial loading failure, background refresh failure, and action failure. Keep last-known detail visible with an explicit stale message and manual Refresh.
- Show **Page updated** separately from **Last evaluated** and evidence receipt time. A successful GET does not prove the worker or server was checked recently. An unchanged event stream does not prove the worker is stalled.
- Keep the existing five-second visibility-aware polling initially, including terminal records. Failed jobs may still have recovery activity; another tab may resume them; later deletion can change identity links and capabilities.
- Use server-loaded initial detail. On route changes and actions, use abortable reads and request generations so old responses cannot replace new detail or feedback. Force reconciliation after an action even if a poll was already in progress.
- Announce meaningful outcome/action changes politely, not every timer tick. Keep textual progress states and `aria-current`; errors remain near their actions. Use a stable loading layout and a clear missing-operation state.
- Reuse existing primitives, spacing, colors, typography, and identity components. Verify long identifiers, keyboard access, visible focus, contrast, reduced motion, 320/390px layouts, tablet/desktop layouts, and 200% zoom.
## Data and implementation design
The page needs structured facts that the current response does not fully expose. Add a small typed read model and observation metadata; preserve the existing lifecycle state machine and execution predicates.
| Contract area | Existing foundation | Required addition |
| --- | --- | --- |
| Identity and outcome | `server_deployments`, historical identity guards | Shared typed detail; recorded result remains distinct from current recovery. |
| Phase and timing | Free-text `stage`, request/finish/submission times | Nullable stable phase key/start time; explicit checkpoint states/times; total elapsed remains based on the original request. |
| Blocker | Worker decision branches and stage messages | Reason code, explanation, evaluated time, automatic continuation condition, and exact dependency reference where known. |
| Command certainty | `submitted_at` is persisted before the network call | Distinguish submission attempted, reply accepted, uncertain outcome, and verified result. Never relabel submission intent as remote acceptance. |
| Verification | Existing probes and transient results | Versioned snapshot with container/game/map/occupancy evidence, result/source/evaluation time, source receipt time where available, and expected/actual values. |
| Recovery and capacity | `restart_rollout`, creation queue, saved concurrency | Exact operation-owned slot/reservation status, relevant limits/holders, and durable recovery/release evidence. |
| Available actions | Existing `canResume*` and `canCancelRemoval` checks | Keep these authoritative; supply contextual explanation without creating new execution capabilities. |
| Provenance and recipe | Sources, collection, saved launch recipe | Named/resolvable context links, missing/deleted-source handling, and explicit queued-capture versus attempted-selection status. |
| Activity | Ordered event ID/time/message | Optional semantic kind/phase/details while preserving existing event text and order. |
A suitable small schema extension is nullable `phase_key`, `phase_started_at`, `observed_at`, and versioned `observation_json` on the operation, plus optional event `kind`, `phase_key`, and allowlisted `details_json`. Final naming can follow repository conventions. Snapshot payloads remain bounded and contain only the fields needed by the page, not raw integration responses or credentials.
Implementation rules:
1. Record structured blockers and observations where the worker already makes decisions. Do not build a second eligibility engine in the API or browser. The detail GET remains read-only and performs no Komodo/RCON probes.
2. Derive current capacity from existing scheduler helpers/tables, with an explicit snapshot time. Persist important milestones against the operation because server-owned recovery rows can later be replaced or deleted. Only attach a live slot when its `operation_id` matches; slot absence alone is not evidence of successful release.
3. Update related state, observation, and semantic event atomically around each durable checkpoint, keeping network calls outside database transactions. Dual-write existing stage/event text, especially `REMOVING_SERVER`, which participates in recovery eligibility today.
4. Record meaningful phase, blocker, command, and recovery transitions. Update observation timestamps on repeated checks without appending an event every poll. Changing player counts must not reset the phase's start time.
5. Preserve prior failure and resume events when an existing operation resumes. New observations must not rewrite the earlier command result or borrow evidence from a newer instance. This does not introduce a new retry engine.
6. For historical records, expose available durable facts and unknowns. Do not backfill invented success timestamps, evidence sources, or completed phases from ambiguous prose.
7. Keep existing IDs, history, API properties, `/api/deployments` compatibility, and legacy redirects. Use additive migrations and test both fresh and upgraded databases. Preserve the legacy `deployment.warnings` string; expose a validated string array through the additive page projection, with an honest fallback for malformed historical values.
Suggested boundaries:
- Shared provisioning detail/progress types near [provisioning.ts](../src/lib/provisioning.ts); a pure presentation mapper for labels, action/state coverage, and legacy fallbacks.
- Server serialization and persistence helpers near [history.ts](../src/lib/server/provisioning/history.ts), with instrumentation in [worker.ts](../src/lib/server/provisioning/worker.ts), [instances.ts](../src/lib/server/provisioning/instances.ts), and [restarts.ts](../src/lib/server/provisioning/restarts.ts).
- An abortable provisioning refresh/action controller under `src/lib/ui/`, taking lessons from the existing metrics refresh controller without changing `pollVisible` globally.
- A thin detail route composed from outcome/actions, execution/evidence, and supporting context. Make activity improvements opt-in or provisioning-specific so campaign/beta history defaults stay intact.
- Reuse the already-fetched detail in [the server loader](../src/routes/provisioning/[id]/+page.server.ts), preserving its campaign fallback and 404 behavior.
The current API returns all events on every poll; the existing Show more limits rendering only. Add an opt-in bounded detail query for this page and its initial server load: five recent entries, event total/latest ID, and the relevant failure excerpt. Fetch full history through `/api/provisioning/:id/events` in 25-entry pages with stable event-ID cursors, literal-text search, and explicit order. Keep existing default detail responses and `/api/deployments` behavior compatible. New events must not shift the active history page; show their count until the reader requests them. Test ordering and cursor stability with concurrent inserts, and measure representative long histories. This is a bounded query change, not a new event-storage system.
## Discussion decisions
| Proposal or disagreement | Decision and reason |
| --- | --- |
| Add a richer dashboard versus reorganize current capabilities | Reorganize around one operation and add only the evidence needed to explain it. Existing recipes, diagnostics, and recovery controls remain useful. |
| Automatically expand all history on failure | Show the failure excerpt and five recent events; keep full expansion under operator control to avoid long, moving pages. |
| Continue fetching all events because only five are displayed | Use an opt-in bounded detail/history API. A five-entry visual preview alone does not bound network or database work, and provisioning history is retained. Preserve legacy consumers. |
| Compress rebuild into a few visual phases | Accepted only if remote deletion, local cleanup, replacement queueing, and verification remain separately inspectable. |
| Improve progress using more stage-string matching | Rejected as the primary model. Use typed worker facts with an honest legacy fallback. Preserve old strings for compatibility and safety predicates. |
| Show an exact queue position or ETA | Show actual blockers/capacity and optionally request order. Admission skips blocked requests; durations are not predictable from rank. |
| Stop refreshing completed/failed jobs | Keep visibility-aware polling initially because recovery, deletion, and permitted actions can still change. |
| Treat Deleted as making every action unavailable | Use backend capabilities. A historical server can still have permitted local cleanup without a live-server link. |
| Treat failed request and recovery as one status | Keep them separate: the historical failure may remain while recovery continues and releases capacity. |
| Add a new event system or scheduler | Rejected. Use additive snapshots and semantic metadata at existing execution boundaries. |
## Delivery sequence
These four increments form the proposed implementation scope. The data contract can be agreed first while layout work proceeds against fixtures.
| Increment | Work | Reviewable result |
| --- | --- | --- |
| 1. Trustworthy operational facts | Shared types, additive migration, worker checkpoint/evidence writes, typed read model, capability and legacy handling. | All six actions provide accurate progress and unknowns without changing execution, command counts, occupancy rules, or slot release. |
| 2. Correct state and refresh behavior | Replace string-derived presentation; fix cancelled/runtime and telemetry copy, deleted cleanup gating, historical outcomes; reuse server-loaded detail; isolate navigation/poll/action requests and feedback. | The existing page remains truthful during cancellation, resume, slow requests, offline refresh, navigation, and continued recovery after failure. |
| 3. Complete detail-page composition | Outcome/action card, blocker/evidence panel, grouped vertical phases, five recent events, bounded full-history queries, metadata/provenance/recipe placement, copy links, return context, responsive/accessibility work. | Operators can find the job's outcome, blocking condition, and valid next action without searching the activity log. |
| 4. Compatibility and release validation | Exercise the full scenario matrix, migration/API compatibility, shared-component regressions, browser behavior, docs, and long-history measurements. | Implementation is ready for normal deployment with documented semantics and evidence of preserved lifecycle guarantees. |
Dependencies: evidence-dependent controls wait for increment 1; refresh and visual work can run in parallel against the agreed contract. Increment 4 is a release gate, not a later cleanup task.
## Acceptance and verification
The review baseline passed **84 tests across five suites**: provisioning, creation queue, operation history, deleted-server guards, and restart rollout. Those tests establish the existing baseline; they do not validate the proposed implementation. No full-build or accessibility-conformance claim is made for this planning work.
Implementation acceptance:
| Scenario | Required result |
| --- | --- |
| Every action across applicable queued/waiting/running/verifying/success/warning/failure/interruption/cancellation states | Correct headline, phase, evidence, timing, and permitted next action. Impossible combinations have explicit legacy/unknown handling. |
| Occupied, unknown, stopped, stale, conflicting RCON/telemetry evidence | UI matches existing eligibility decisions; positive occupancy blocks disruption; stale/pre-submission/future telemetry cannot masquerade as accepted evidence. |
| Capacity or parent/dependency hold | Actual blocker and matching links appear; no guaranteed admission order, fabricated check time, or ETA. |
| Ambiguous remote reply, then recovery | No duplicate external command; slot stays reserved until the existing rules release/transfer it; historical failure and recovery evidence remain distinguishable. |
| Partial destroy/rebuild and deleted identity | Confirmed remote deletion is retained; local cleanup can resume only when permitted; creation waits for full cleanup; old work never targets a replacement instance. |
| Cancellation or resume racing a worker claim/newer operation | Backend remains authoritative; 409 reconciles the page; no automatic mutation replay or false success. |
| Resume of the same operation ID | Prior failure outcome/time and explicit resume event remain preserved in semantic history while current progress/finish fields update; later completion records its own outcome without a new retry system. |
| Telemetry success / wrong map / historical success | Evidence source and timestamps are accurate; expected and observed engine maps are clear; the page does not claim current health from old results. |
| Slow poll + action, A→B→A navigation, stale error, unmount | Obsolete responses cannot affect current detail or action feedback; successful mutations invalidate stale reads and trigger reconciliation; pending state resets appropriately. If reconciliation fails, action acknowledgment and stale-data feedback remain distinct. |
| Refresh failure and recovery | Last snapshot remains visible with stale labeling; Retry/Refresh works; action errors are not silently erased. |
| Long activity, new events while reading | Search/order/disclosures/focus remain stable; new-event indicator works; no forced scrolling. Responses stay bounded; cursor pages have no duplicates/gaps within their documented ordering; legacy defaults remain compatible. Measure payload/query/render cost. |
| Promotion occurs while create/rebuild is queued | Queued recipe is labeled as a capture, final attempted selection is recorded at creation, and the page does not imply pinned versions or successful installation. |
| Legacy DB, absent recipe/evidence/source, deleted/reused name | Honest missing-data presentation, preserved IDs/events/foreign keys, safe links/capabilities, and unchanged legacy route/API compatibility. |
| Keyboard, assistive announcements, phone, zoom, long text | No loss of reachable actions or page-wide overflow; progress is understandable without color; timers do not cause repeated announcements. |
Extend behavior tests around these risks rather than snapshotting markup or mirroring DTO construction. Verify that detail GETs perform no external probes. Run the affected lifecycle/history/refresh tests, then the full suite, `npm run check`, and `npm run build` for the implementation. Validate desktop/mobile browser fixtures and update [Provisioning documentation](provisioning.md) and relevant README descriptions.
Preserve the existing execution rules throughout: fresh occupancy gates, startup/restart limits, ambiguous-response handling, historical identity checks, exact cleanup checkpoints, requester ownership, and observation-only resumes. Normal deployment should sample both a newly instrumented operation and a legacy record; presentation rollback must not require deleting the new history.
Outside this proposal: generic retry/force-run controls, new cancellation semantics, scheduler redesign, real-time transport migration, a general observability platform, broad redesigns of other operation pages, and predicted completion times.
Binary file not shown.

After

Width:  |  Height:  |  Size: 202 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 130 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 424 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 358 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 295 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 357 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 200 KiB

+63
View File
@@ -75,6 +75,59 @@ new attempt, preserving prior history; paused or closed beta work cannot start a
queued restart. Recovery of an already submitted restart continues even if its queued restart. Recovery of an already submitted restart continues even if its
requester closes. Upgrades adopt existing restart intent for observation only. requester closes. Upgrades adopt existing restart intent for observation only.
## Operation details
Each job page puts its recorded outcome, blocking condition, and available next
action before launch files. Execution is grouped into phases, with separate
milestones for remote deletion, local cleanup, and replacement creation during a
rebuild. Copy the operation ID or link when investigating or handing off a job.
Related work links back to the requesting campaign, beta, maintenance item, or
collection. A completed provisioning job does not imply that its requester has
finished its own verification.
New operations record structured verification evidence alongside their activity:
the check result, source, evaluation time, and expected/observed map when relevant.
Game status can be verified by RCON or accepted fresh telemetry. Engine-map checks
do not independently verify a T6 start location. Passed checks describe their
recorded observation; later checks can fail or become unknown. Completion describes
what was verified at that time, while **View server** opens the current instance.
Older history retains its recorded stages and outcomes; unavailable evidence is
not inferred from stage wording.
Submission attempted, command accepted, and verified completion are separate
milestones. A failed restart may still have ongoing recovery observation and hold
restart capacity. Its original failure remains in history even if recovery later
succeeds and releases the slot. Capacity and request order are snapshots, not
guarantees of the next job admitted or an estimated completion time. Blocked
requests can be bypassed by eligible work.
Recovery controls use the server's current eligibility checks. **Resume health
verification** observes the existing instance without submitting another create or
restart. **Resume cleanup** retries local cleanup after confirmed remote deletion,
including eligible historical deleted identities; replacement or newer-operation
guards still apply. **Resume rebuild cleanup** completes cleanup before creating
the replacement in its reserved slot. Resuming the same operation preserves its
earlier failure and resume events. Uncertain commands have no generic Retry action.
Create and rebuild capture a setup when queued and select the latest promoted
files again when creation begins. The page distinguishes **Setup captured when
queued** from **Files selected for this attempt**. A queued capture does not pin
versions; an attempted selection is not proof that every file was installed.
**Launch another server** uses the latest promoted setup with a fresh identity and
port, rather than repairing or replaying the recorded job.
Five recent events remain visible. Expand full history to search, change order,
and load older pages. A fixed history watermark keeps the viewed pages stable as
events arrive; **Jump to latest** incorporates new activity. The detail view
fetches a bounded recent-event window, while full history loads on demand.
The page refreshes every five seconds while visible, including terminal jobs whose
recovery or available actions may still change. **Page updated** describes the
last successful detail fetch; it is separate from the worker's evaluation and
telemetry receipt times. A refresh failure keeps the last snapshot visible and
does not erase an acknowledged action result. These timestamps are not worker
heartbeat or completion-time promises.
## Routes and compatibility ## Routes and compatibility
- `/provisioning`: active operations and history filtered by action. - `/provisioning`: active operations and history filtered by action.
@@ -89,6 +142,16 @@ requester closes. Upgrades adopt existing restart intent for observation only.
- `/api/provisioning/:id`, `/:id/resume`, `/:id/cancel`, `/preview`, and - `/api/provisioning/:id`, `/:id/resume`, `/:id/cancel`, `/preview`, and
`/house-party` preserve the existing operation, recovery, removal cancellation, `/house-party` preserve the existing operation, recovery, removal cancellation,
launch preview, and fleet destruction APIs. launch preview, and fleet destruction APIs.
- `GET /api/provisioning/:id?events=recent` returns the same detail with five recent
events plus activity totals/latest ID and a failure excerpt. Omitting the option
preserves the full legacy event response. Progress, capacity, named related work,
and normalized warning messages are additive fields; the original warning string
and lifecycle fields remain available.
- `GET /api/provisioning/:id/events` reads history with `order=newest|oldest`,
literal-text `query`, `limit` (default 25, maximum 100), an exclusive event-ID
`cursor`, and an upper-ID `watermark`. Keep the returned watermark when loading
more and use `nextCursor` until it is null. Begin without a watermark to include
newly arrived events. These reads do not probe Komodo or RCON.
Old `/deployments` pages redirect to their `/provisioning` equivalents, preserving Old `/deployments` pages redirect to their `/provisioning` equivalents, preserving
query parameters. Collection launch links at `/provisioning/collections/:id` or query parameters. Collection launch links at `/provisioning/collections/:id` or
+121
View File
@@ -0,0 +1,121 @@
import type { ProvisioningAction } from './provisioning.js';
import type { LaunchRecipeSummary } from './deployment-plan.js';
export const provisioningPhases = ['requested', 'eligibility', 'startup-queue', 'prepare', 'create-deployment',
'container-action', 'remove-container', 'delete-deployment', 'cleanup', 'verify-container', 'verify-game',
'verify-map', 'verify-stopped'] as const;
export type ProvisioningPhase = typeof provisioningPhases[number];
export type EvidenceResult = 'pass' | 'waiting' | 'fail' | 'unknown';
export type EvidenceSource = 'komodo' | 'rcon' | 'telemetry' | 'combined' | null;
export interface ProvisioningCheck {
key: 'container' | 'game' | 'map' | 'occupancy' | 'stopped';
label: string;
result: EvidenceResult;
source: EvidenceSource;
checkedAt: string;
receivedAt: string | null;
expected: string | number | null;
actual: string | number | null;
message: string | null;
}
export interface ProvisioningBlocker {
code: string;
message: string;
automatic: boolean;
href?: string | null;
label?: string | null;
}
export interface ProvisioningCheckpoint {
key: string;
label: string;
at: string;
phase: ProvisioningPhase | null;
}
export interface RecoveryObservation {
state: 'held' | 'released' | 'transferred' | 'unknown';
at: string;
message: string;
operationId?: string | null;
}
export interface ProvisioningProgress {
version: 1;
phase: ProvisioningPhase | null;
phaseStartedAt: string | null;
observedAt: string | null;
blocker: ProvisioningBlocker | null;
checks: ProvisioningCheck[];
checkpoints: ProvisioningCheckpoint[];
command: { state: 'attempted' | 'accepted' | 'rejected' | 'uncertain'; at: string; message: string } | null;
recovery: RecoveryObservation | null;
recipeStatus: 'queued' | 'attempted' | null;
}
export interface ProvisioningEvent {
id: number;
created_at: string;
message: string;
kind?: string | null;
phase_key?: string | null;
details_json?: string | null;
}
export interface ProvisioningDeployment {
id: string;
server: string;
server_deleted_at: string | null;
action: ProvisioningAction;
state: string;
stage: string;
started_at: string;
finished_at: string | null;
submitted_at?: string | null;
checked_at?: string | null;
players?: number | null;
triggered_by: string;
error: string | null;
warnings: string;
server_exists: number;
expected_map?: string | null;
}
export interface ProvisioningSource {
kind: 'campaign' | 'beta' | 'maintenance' | 'watchdog';
source_id: string;
server?: string;
label: string;
href: string | null;
}
export interface ProvisioningCapacity {
evaluatedAt: string;
pools: {
kind: 'startup' | 'restart';
limit: number;
used: number;
holders: { id: string | null; server: string }[];
requestOrder: number | null;
ownsSlot: boolean;
}[];
}
export interface ProvisioningDetail {
deployment: ProvisioningDeployment;
events: ProvisioningEvent[];
activity: { total: number; latestId: number | null; failureExcerpt: ProvisioningEvent | null };
sources: ProvisioningSource[];
recipe: LaunchRecipeSummary | null;
blueprintName: string | null;
canResumeVerification: boolean;
canResumeCleanup: boolean;
canResumeRebuild: boolean;
canCancelRemoval: boolean;
collection: { id: string; name: string } | null;
progress: ProvisioningProgress | null;
capacity: ProvisioningCapacity;
warningMessages: string[];
generatedAt: string;
}
export interface ProvisioningEventPage {
events: ProvisioningEvent[];
total: number;
totalEvents: number;
latestId: number | null;
watermark: number;
nextCursor: number | null;
}
+14 -3
View File
@@ -243,8 +243,10 @@ function migrateProvisioningActions(handle: Database.Database): void {
handle.transaction(() => { handle.transaction(() => {
handle.exec(table.sql.replace('server_deployments', 'server_deployments_new') handle.exec(table.sql.replace('server_deployments', 'server_deployments_new')
.replace("'create', 'destroy'", "'create', 'start', 'restart', 'rebuild', 'stop', 'destroy'")); .replace("'create', 'destroy'", "'create', 'start', 'restart', 'rebuild', 'stop', 'destroy'"));
handle.exec(`INSERT INTO server_deployments_new (rowid, id, server, action, state, stage, started_at, finished_at, triggered_by, error, warnings) const columns = (handle.prepare('PRAGMA table_info(server_deployments)').all() as { name: string }[])
SELECT rowid, id, server, action, state, stage, started_at, finished_at, triggered_by, error, warnings FROM server_deployments; .map(column => `"${column.name.replaceAll('"', '""')}"`).join(', ');
handle.exec(`INSERT INTO server_deployments_new (rowid, ${columns})
SELECT rowid, ${columns} FROM server_deployments;
DROP TABLE server_deployments; DROP TABLE server_deployments;
ALTER TABLE server_deployments_new RENAME TO server_deployments;`); ALTER TABLE server_deployments_new RENAME TO server_deployments;`);
if ((handle.pragma('foreign_key_check') as unknown[]).length) throw new Error('Provisioning migration found broken references'); if ((handle.pragma('foreign_key_check') as unknown[]).length) throw new Error('Provisioning migration found broken references');
@@ -309,7 +311,16 @@ export function db(): Database.Database {
migrateTable(handle, 'server_deployments', [ migrateTable(handle, 'server_deployments', [
{ name: 'submitted_at', ddl: 'ALTER TABLE server_deployments ADD COLUMN submitted_at TEXT' }, { name: 'submitted_at', ddl: 'ALTER TABLE server_deployments ADD COLUMN submitted_at TEXT' },
{ name: 'checked_at', ddl: 'ALTER TABLE server_deployments ADD COLUMN checked_at TEXT' }, { name: 'checked_at', ddl: 'ALTER TABLE server_deployments ADD COLUMN checked_at TEXT' },
{ name: 'players', ddl: 'ALTER TABLE server_deployments ADD COLUMN players INTEGER' } { name: 'players', ddl: 'ALTER TABLE server_deployments ADD COLUMN players INTEGER' },
{ name: 'phase_key', ddl: 'ALTER TABLE server_deployments ADD COLUMN phase_key TEXT' },
{ name: 'phase_started_at', ddl: 'ALTER TABLE server_deployments ADD COLUMN phase_started_at TEXT' },
{ name: 'observed_at', ddl: 'ALTER TABLE server_deployments ADD COLUMN observed_at TEXT' },
{ name: 'observation_json', ddl: 'ALTER TABLE server_deployments ADD COLUMN observation_json TEXT' }
]);
migrateTable(handle, 'server_deployment_events', [
{ name: 'kind', ddl: 'ALTER TABLE server_deployment_events ADD COLUMN kind TEXT' },
{ name: 'phase_key', ddl: 'ALTER TABLE server_deployment_events ADD COLUMN phase_key TEXT' },
{ name: 'details_json', ddl: 'ALTER TABLE server_deployment_events ADD COLUMN details_json TEXT' }
]); ]);
for (const table of ['deployment_servers', 'beta_servers', 'maintenance_jobs']) { for (const table of ['deployment_servers', 'beta_servers', 'maintenance_jobs']) {
migrateTable(handle, table, [{ name: 'provisioning_id', ddl: `ALTER TABLE ${table} ADD COLUMN provisioning_id TEXT` }]); migrateTable(handle, table, [{ name: 'provisioning_id', ddl: `ALTER TABLE ${table} ADD COLUMN provisioning_id TEXT` }]);
+9 -2
View File
@@ -452,14 +452,21 @@ CREATE TABLE IF NOT EXISTS server_deployments (
finished_at TEXT, finished_at TEXT,
triggered_by TEXT NOT NULL, triggered_by TEXT NOT NULL,
error TEXT, error TEXT,
warnings TEXT NOT NULL DEFAULT '[]' warnings TEXT NOT NULL DEFAULT '[]',
phase_key TEXT,
phase_started_at TEXT,
observed_at TEXT,
observation_json TEXT
); );
CREATE TABLE IF NOT EXISTS server_deployment_events ( CREATE TABLE IF NOT EXISTS server_deployment_events (
id INTEGER PRIMARY KEY AUTOINCREMENT, id INTEGER PRIMARY KEY AUTOINCREMENT,
deployment_id TEXT NOT NULL REFERENCES server_deployments(id) ON DELETE CASCADE, deployment_id TEXT NOT NULL REFERENCES server_deployments(id) ON DELETE CASCADE,
created_at TEXT NOT NULL, created_at TEXT NOT NULL,
message TEXT NOT NULL message TEXT NOT NULL,
kind TEXT,
phase_key TEXT,
details_json TEXT
); );
CREATE INDEX IF NOT EXISTS idx_server_deployment_events ON server_deployment_events(deployment_id, id); CREATE INDEX IF NOT EXISTS idx_server_deployment_events ON server_deployment_events(deployment_id, id);
+27 -9
View File
@@ -36,30 +36,48 @@ export function telemetryEvidence(server: string, after: string | null = latestG
/** Positive occupancy from either source always vetoes an empty-server claim. */ /** Positive occupancy from either source always vetoes an empty-server claim. */
export async function probePlayerCount(server: string): Promise<number | null> { export async function probePlayerCount(server: string): Promise<number | null> {
return (await probePlayerEvidence(server)).players;
}
export interface PlayerEvidence {
players: number | null;
source: 'rcon' | 'telemetry' | null;
observedAt: string;
receivedAt?: string;
}
/** The same eligibility decision, with provenance from the probes already performed. */
export async function probePlayerEvidence(server: string): Promise<PlayerEvidence> {
const fromTelemetry = (value: TelemetryEvidence): PlayerEvidence => ({
players: value.players, source: 'telemetry', observedAt: new Date().toISOString(), receivedAt: value.receivedAt
});
const occupied = telemetryEvidence(server); const occupied = telemetryEvidence(server);
if (occupied && occupied.players > 0) return occupied.players; if (occupied && occupied.players > 0) return fromTelemetry(occupied);
const players = await rconGetPlayerCount(server).catch(() => null); const players = await rconGetPlayerCount(server).catch(() => null);
// Read after the network request: a player may join while RCON is pending. // Read after the network request: a player may join while RCON is pending.
const telemetry = telemetryEvidence(server); const telemetry = telemetryEvidence(server);
if (players !== null && players > 0) return players; if (players !== null && players > 0) return { players, source: 'rcon', observedAt: new Date().toISOString() };
if (telemetry && telemetry.players > 0) return telemetry.players; if (telemetry && telemetry.players > 0) return fromTelemetry(telemetry);
return players === 0 || telemetry?.players === 0 ? 0 : null; if (players === 0) return { players, source: 'rcon', observedAt: new Date().toISOString() };
if (telemetry?.players === 0) return fromTelemetry(telemetry);
return { players: null, source: null, observedAt: new Date().toISOString() };
} }
/** Komodo readiness remains a caller prerequisite. Old-process telemetry is excluded. */ /** Komodo readiness remains a caller prerequisite. Old-process telemetry is excluded. */
export async function probeServerStatus(server: string, after?: string | null): Promise< export async function probeServerStatus(server: string, after?: string | null): Promise<
{ ok: true; players: number; map: string | null; source: 'rcon' | 'telemetry' } | { ok: false; reason: string } { ok: true; players: number; map: string | null; source: 'rcon' | 'telemetry'; observedAt: string; receivedAt?: string } | { ok: false; reason: string; observedAt: string }
> { > {
const current = telemetryEvidence(server, after); const current = telemetryEvidence(server, after);
if (current) return { ok: true, ...current, source: 'telemetry' }; if (current) return { ok: true, ...current, source: 'telemetry', observedAt: new Date().toISOString() };
let reason = 'RCON unavailable'; let reason = 'RCON unavailable';
try { try {
const rcon = await rconStatusResult(server); const rcon = await rconStatusResult(server);
const latest = telemetryEvidence(server, after); const latest = telemetryEvidence(server, after);
if (latest) return { ok: true, ...latest, source: 'telemetry' }; if (latest) return { ok: true, ...latest, source: 'telemetry', observedAt: new Date().toISOString() };
if (rcon.ok) return { ok: true, players: rcon.status.players, map: rcon.status.map, source: 'rcon' }; if (rcon.ok) return { ok: true, players: rcon.status.players, map: rcon.status.map, source: 'rcon', observedAt: new Date().toISOString() };
reason = rcon.reason; reason = rcon.reason;
} catch (err) { reason = (err as Error).message; } } catch (err) { reason = (err as Error).message; }
const telemetry = telemetryEvidence(server, after); const telemetry = telemetryEvidence(server, after);
return telemetry ? { ok: true, ...telemetry, source: 'telemetry' } : { ok: false, reason }; return telemetry ? { ok: true, ...telemetry, source: 'telemetry', observedAt: new Date().toISOString() }
: { ok: false, reason, observedAt: new Date().toISOString() };
} }
+56
View File
@@ -0,0 +1,56 @@
import { db } from '../db/index.js';
import type { ProvisioningEvent, ProvisioningEventPage } from '../../provisioning-detail.js';
type ActivityRow = ProvisioningEvent;
export interface ActivityQuery {
order?: 'newest' | 'oldest';
query?: string;
cursor?: number;
watermark?: number;
limit?: number;
}
export function activitySummary(id: string): { total: number; latestId: number | null } {
return db().prepare(`SELECT COUNT(*) AS total, MAX(id) AS latestId
FROM server_deployment_events WHERE deployment_id = ?`).get(id) as { total: number; latestId: number | null };
}
/** ID ordering and a fixed upper watermark keep older pages stable as new events arrive. */
export function deploymentActivity(id: string, options: ActivityQuery = {}): ProvisioningEventPage {
const summary = activitySummary(id);
const watermark = Math.min(options.watermark ?? summary.latestId ?? 0, summary.latestId ?? 0);
const limit = Math.max(1, Math.min(100, Math.trunc(options.limit ?? 25)));
const newest = options.order !== 'oldest';
const query = options.query ?? '';
const base = 'deployment_id = ? AND id <= ? AND instr(lower(message), lower(?)) > 0';
const values: (string | number)[] = [id, watermark, query];
const total = (db().prepare(`SELECT COUNT(*) AS n FROM server_deployment_events WHERE ${base}`)
.get(...values) as { n: number }).n;
const cursor = options.cursor === undefined ? '' : ` AND id ${newest ? '<' : '>'} ?`;
if (options.cursor !== undefined) values.push(options.cursor);
const rows = db().prepare(`SELECT id, created_at, message, kind, phase_key, details_json
FROM server_deployment_events WHERE ${base}${cursor}
ORDER BY id ${newest ? 'DESC' : 'ASC'} LIMIT ?`).all(...values, limit + 1) as ActivityRow[];
const more = rows.length > limit;
const events = rows.slice(0, limit);
return { events, total, totalEvents: summary.total, latestId: summary.latestId, watermark,
nextCursor: more ? events.at(-1)!.id : null };
}
export function deploymentEvents(id: string, recent = false): ProvisioningEvent[] {
if (recent) return (db().prepare(`SELECT id, created_at, message, kind, phase_key, details_json
FROM server_deployment_events WHERE deployment_id = ? ORDER BY id DESC LIMIT 5`).all(id) as ActivityRow[]).reverse();
return (db().prepare(`SELECT id, created_at, message, kind, phase_key, details_json
FROM server_deployment_events WHERE deployment_id = ? ORDER BY id`).all(id) as ActivityRow[]);
}
export function latestFailureEvent(id: string): ProvisioningEvent | null {
const row = db().prepare(`SELECT id, created_at, message, kind, phase_key, details_json
FROM server_deployment_events WHERE deployment_id = ?
AND (kind IN ('failed', 'interrupted') OR (kind = 'outcome'
AND CASE WHEN json_valid(details_json) THEN json_extract(details_json, '$.state') END IN ('failed', 'interrupted'))
OR (kind IS NULL AND (message LIKE 'Failed:%' OR message LIKE 'Interrupted by application restart.%')))
ORDER BY id DESC LIMIT 1`).get(id) as ActivityRow | undefined;
return row ?? null;
}
+139 -26
View File
@@ -2,20 +2,35 @@ import { randomUUID } from 'node:crypto';
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import { recipeForDeployment, summarizeRecipe } from '../fleet/launch-recipes.js'; import { recipeForDeployment, summarizeRecipe } from '../fleet/launch-recipes.js';
import { actionLabels, type ProvisioningAction } from '../../provisioning.js'; import { actionLabels, type ProvisioningAction } from '../../provisioning.js';
import type { ProvisioningCapacity, ProvisioningDeployment, ProvisioningDetail, ProvisioningSource } from '../../provisioning-detail.js';
import { fleetSettings } from '../settings.js';
import { activitySummary, deploymentEvents, latestFailureEvent } from './activity.js';
import { readProgress, recordProgress, type ProgressUpdate } from './progress.js';
export function beginDeployment(server: string, action: ProvisioningAction, triggeredBy = 'ui'): string { export function beginDeployment(server: string, action: ProvisioningAction, triggeredBy = 'ui'): string {
const id = randomUUID(); const id = randomUUID();
db().transaction(() => {
db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by) db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by)
VALUES (?, ?, ?, 'running', 'Starting', ?, ?)`).run(id, server, action, new Date().toISOString(), triggeredBy); VALUES (?, ?, ?, 'running', 'Starting', ?, ?)`).run(id, server, action, new Date().toISOString(), triggeredBy);
recordDeploymentEvent(id, action === 'create' ? 'Server creation requested' recordProgress(id, { phase: 'requested', observedAt: null, checkpoint: { key: 'requested', phase: 'requested',
: action === 'destroy' ? 'Server destruction requested' : `${actionLabels[action]} server requested`); label: action === 'create' ? 'Server creation requested'
: action === 'destroy' ? 'Server destruction requested' : `${actionLabels[action]} server requested` } });
})();
return id; return id;
} }
export function deploymentStage(id: string, stage: string): void { export function deploymentStage(id: string, stage: string, update?: ProgressUpdate): void {
db().transaction(() => {
const previous = db().prepare('SELECT stage FROM server_deployments WHERE id = ?').get(id) as { stage: string } | undefined; const previous = db().prepare('SELECT stage FROM server_deployments WHERE id = ?').get(id) as { stage: string } | undefined;
const changed = !!previous && previous.stage !== stage;
const before = update && changed ? activitySummary(id).latestId ?? 0 : 0;
db().prepare('UPDATE server_deployments SET stage = ? WHERE id = ?').run(stage, id); db().prepare('UPDATE server_deployments SET stage = ? WHERE id = ?').run(stage, id);
if (previous && previous.stage !== stage) recordDeploymentEvent(id, stage); if (update) recordProgress(id, update);
// Structured transitions may already carry the exact legacy stage text.
const alreadyRecorded = update && changed && db().prepare(`SELECT 1 FROM server_deployment_events
WHERE deployment_id = ? AND id > ? AND message = ? LIMIT 1`).get(id, before, stage);
if (changed && !alreadyRecorded) recordDeploymentEvent(id, stage);
})();
} }
export function recordDeploymentEvent(id: string, message: string): void { export function recordDeploymentEvent(id: string, message: string): void {
@@ -23,19 +38,83 @@ export function recordDeploymentEvent(id: string, message: string): void {
.run(id, new Date().toISOString(), message); .run(id, new Date().toISOString(), message);
} }
export function getDeployment(id: string) { function warningMessages(raw: string): string[] {
const deployment = db().prepare(`SELECT d.*, j.expected_map, (d.server_deleted_at IS NULL AND EXISTS(SELECT 1 FROM servers s WHERE s.name = d.server AND s.destroyed_at IS NULL)) AS server_exists try {
FROM server_deployments d LEFT JOIN server_creation_jobs j ON j.deployment_id = d.id WHERE d.id = ?`).get(id); const parsed: unknown = JSON.parse(raw);
if (!deployment) return null; if (!Array.isArray(parsed)) return ['Recorded warnings could not be read.'];
const events = db().prepare('SELECT id, created_at, message FROM server_deployment_events WHERE deployment_id = ? ORDER BY id').all(id); const messages = parsed.filter((value): value is string => typeof value === 'string');
const sources = db().prepare('SELECT kind, source_id, server FROM provisioning_sources WHERE operation_id = ? ORDER BY kind, source_id').all(id); if (messages.length !== parsed.length) messages.push('Some recorded warnings could not be read.');
return messages;
} catch { return ['Recorded warnings could not be read.']; }
}
function sourceDetails(source: Omit<ProvisioningSource, 'label' | 'href'>): ProvisioningSource {
const id = source.source_id;
if (source.kind === 'campaign') {
const row = db().prepare('SELECT action FROM deployments WHERE id = ?').get(id) as { action: string } | undefined;
const action = row?.action === 'promote' ? 'Promotion' : row?.action === 'rollback' ? 'Rollback' : row?.action === 'drift' ? 'Reconciliation' : 'Campaign';
return { ...source, label: `${action} ${id.slice(0, 8)}`, href: row ? `/campaigns/${encodeURIComponent(id)}` : null };
}
if (source.kind === 'beta') {
const row = db().prepare('SELECT notes FROM betas WHERE id = ?').get(id) as { notes: string | null } | undefined;
const label = row?.notes?.trim().split('\n')[0].slice(0, 120);
return { ...source, label: label ? `Beta #${id}: ${label}` : `Beta #${id}`, href: row ? `/betas/${encodeURIComponent(id)}` : null };
}
if (source.kind === 'watchdog') {
const row = db().prepare('SELECT summary FROM watchdog_incidents WHERE id = ?').get(id) as { summary: string } | undefined;
return { ...source, label: row?.summary ?? `Maintenance finding ${id.slice(0, 8)}`, href: row ? `/maintenance/${encodeURIComponent(id)}` : null };
}
const row = db().prepare('SELECT kind FROM maintenance_jobs WHERE id = ?').get(id) as { kind: string } | undefined;
return { ...source, label: row && row.kind !== 'legacy' ? `Maintenance ${row.kind} #${id}` : `Maintenance #${id}`,
href: row ? `/maintenance?view=activity#job-${encodeURIComponent(id)}` : null };
}
/** These are current pool snapshots, not a second admission/eligibility engine. */
function capacityFor(deployment: ProvisioningDeployment, evaluatedAt: string): ProvisioningCapacity {
const settings = fleetSettings();
const pools: ProvisioningCapacity['pools'] = [];
if (deployment.action === 'create' || deployment.action === 'rebuild') {
const jobs = db().prepare(`SELECT id, server, state FROM server_deployments
WHERE server_deleted_at IS NULL AND action IN ('create','rebuild') AND state IN ('queued','running','awaiting-health')
ORDER BY started_at, rowid`).all() as { id: string; server: string; state: string }[];
const holders = jobs.filter(job => job.state !== 'queued').map(({ id, server }) => ({ id, server }));
const order = jobs.filter(job => job.state === 'queued').findIndex(job => job.id === deployment.id);
pools.push({ kind: 'startup', limit: settings.createConcurrency, used: holders.length, holders,
requestOrder: order < 0 ? null : order + 1, ownsSlot: holders.some(holder => holder.id === deployment.id) });
}
if (['start', 'restart', 'rebuild'].includes(deployment.action)) {
const holders = db().prepare(`SELECT operation_id AS id, server FROM restart_rollout WHERE healthy_at IS NULL
ORDER BY requested_at, server`).all() as { id: string | null; server: string }[];
pools.push({ kind: 'restart', limit: settings.restartConcurrency, used: holders.length, holders,
requestOrder: null, ownsSlot: holders.some(holder => holder.id === deployment.id) });
}
return { evaluatedAt, pools };
}
/** The default retains the full legacy event response. Page clients opt into a bounded window. */
export function getDeployment(id: string, options: { events?: 'all' | 'recent' } = {}): ProvisioningDetail | null {
return db().transaction(() => {
const row = db().prepare(`SELECT d.*, j.expected_map, (d.server_deleted_at IS NULL AND EXISTS(SELECT 1 FROM servers s WHERE s.name = d.server AND s.destroyed_at IS NULL)) AS server_exists
FROM server_deployments d LEFT JOIN server_creation_jobs j ON j.deployment_id = d.id WHERE d.id = ?`).get(id) as
(ProvisioningDeployment & { observation_json: string | null; phase_key: string | null; phase_started_at: string | null; observed_at: string | null }) | undefined;
if (!row) return null;
const generatedAt = new Date().toISOString();
const { observation_json: _snapshot, phase_key: _phase, phase_started_at: _phaseStarted, observed_at: _observed, ...deployment } = row;
const sources = (db().prepare('SELECT kind, source_id, server FROM provisioning_sources WHERE operation_id = ? ORDER BY kind, source_id')
.all(id) as Omit<ProvisioningSource, 'label' | 'href'>[]).map(sourceDetails);
const launch = recipeForDeployment(id); const launch = recipeForDeployment(id);
const collection = db().prepare(`SELECT c.id, c.name FROM ( const collection = db().prepare(`SELECT c.id, c.name FROM (
SELECT collection_id, deployment_id FROM server_collection_members SELECT collection_id, deployment_id FROM server_collection_members
UNION ALL SELECT collection_id, deployment_id FROM server_collection_removals) m UNION ALL SELECT collection_id, deployment_id FROM server_collection_removals) m
JOIN server_launch_collections c ON c.id = m.collection_id WHERE m.deployment_id = ?`).get(id) ?? null; JOIN server_launch_collections c ON c.id = m.collection_id WHERE m.deployment_id = ? ORDER BY c.created_at, c.id LIMIT 1`).get(id) as { id: string; name: string } | undefined;
return { deployment, events, sources, recipe: launch ? summarizeRecipe(launch.recipe) : null, blueprintName: launch?.blueprintName ?? null, return { deployment, events: deploymentEvents(id, options.events === 'recent'),
canResumeVerification: canResumeVerification(id), canResumeCleanup: canResumeCleanup(id), canResumeRebuild: canResumeRebuild(id), canCancelRemoval: canCancelRemoval(id), collection }; activity: { ...activitySummary(id), failureExcerpt: latestFailureEvent(id) }, sources,
recipe: launch ? summarizeRecipe(launch.recipe) : null, blueprintName: launch?.blueprintName ?? null,
canResumeVerification: canResumeVerification(id), canResumeCleanup: canResumeCleanup(id),
canResumeRebuild: canResumeRebuild(id), canCancelRemoval: canCancelRemoval(id), collection: collection ?? null,
progress: readProgress(row), capacity: capacityFor(deployment, generatedAt),
warningMessages: warningMessages(deployment.warnings), generatedAt };
})();
} }
// This stage is written only after Komodo has confirmed deployment deletion. // This stage is written only after Komodo has confirmed deployment deletion.
@@ -48,10 +127,13 @@ export function canCancelRemoval(id: string): boolean {
/** Competes atomically with the worker's claim, before any external removal. */ /** Competes atomically with the worker's claim, before any external removal. */
export function cancelRemoval(id: string): boolean { export function cancelRemoval(id: string): boolean {
return db().transaction(() => { return db().transaction(() => {
const finishedAt = new Date().toISOString();
const result = db().prepare(`UPDATE server_deployments SET state = 'cancelled', stage = 'Removal cancelled', finished_at = ?, error = NULL const result = db().prepare(`UPDATE server_deployments SET state = 'cancelled', stage = 'Removal cancelled', finished_at = ?, error = NULL
WHERE id = ? AND server_deleted_at IS NULL AND action = 'destroy' AND state = 'waiting' AND stage != ?`).run(new Date().toISOString(), id, REMOVING_SERVER); WHERE id = ? AND server_deleted_at IS NULL AND action = 'destroy' AND state = 'waiting' AND stage != ?`).run(finishedAt, id, REMOVING_SERVER);
if (!result.changes) return false; if (!result.changes) return false;
recordDeploymentEvent(id, 'Queued removal cancelled. The server and its files were kept.'); recordProgress(id, { blocker: null, checks: [], event: { kind: 'outcome',
message: 'Queued removal cancelled. The server and its files were kept.',
details: { state: 'cancelled', finishedAt } } });
return true; return true;
})(); })();
} }
@@ -76,8 +158,10 @@ export function canResumeCleanup(id: string): boolean {
export function resumeCleanup(id: string): boolean { export function resumeCleanup(id: string): boolean {
return db().transaction(() => { return db().transaction(() => {
if (!canResumeCleanup(id)) return false; if (!canResumeCleanup(id)) return false;
preserveOutcome(id);
db().prepare("UPDATE server_deployments SET state = 'waiting', finished_at = NULL, error = NULL WHERE id = ?").run(id); db().prepare("UPDATE server_deployments SET state = 'waiting', finished_at = NULL, error = NULL WHERE id = ?").run(id);
recordDeploymentEvent(id, 'Resumed local cleanup after confirmed Komodo deletion'); recordProgress(id, { phase: 'cleanup', blocker: null, checks: [], observedAt: null,
event: { kind: 'resumed', message: 'Resumed local cleanup after confirmed Komodo deletion' } });
return true; return true;
})(); })();
} }
@@ -100,29 +184,55 @@ export function canResumeRebuild(id: string): boolean {
} }
export function resumeRebuild(id: string): boolean { export function resumeRebuild(id: string): boolean {
return db().transaction(() => {
if (!canResumeRebuild(id)) return false; if (!canResumeRebuild(id)) return false;
preserveOutcome(id);
db().prepare("UPDATE server_deployments SET state = 'waiting', finished_at = NULL, error = NULL WHERE id = ?").run(id); db().prepare("UPDATE server_deployments SET state = 'waiting', finished_at = NULL, error = NULL WHERE id = ?").run(id);
recordDeploymentEvent(id, 'Resumed rebuild cleanup after confirmed Komodo deletion'); recordProgress(id, { phase: 'cleanup', blocker: null, checks: [], observedAt: null,
event: { kind: 'resumed', message: 'Resumed rebuild cleanup after confirmed Komodo deletion' } });
return true; return true;
})();
} }
/** Resume observation only. Never repeat a container creation or restart. */ /** Resume observation only. Never repeat a container creation or restart. */
export function resumeVerification(id: string): boolean { export function resumeVerification(id: string): boolean {
return db().transaction(() => { return db().transaction(() => {
if (!canResumeVerification(id)) return false; if (!canResumeVerification(id)) return false;
preserveOutcome(id);
db().prepare("UPDATE server_deployments SET state = 'awaiting-health', finished_at = NULL, error = NULL WHERE id = ?").run(id); db().prepare("UPDATE server_deployments SET state = 'awaiting-health', finished_at = NULL, error = NULL WHERE id = ?").run(id);
recordDeploymentEvent(id, 'Resumed verification of the existing server'); deploymentStage(id, 'Waiting for Komodo health', { phase: 'verify-container', blocker: null, checks: [], observedAt: null,
deploymentStage(id, 'Waiting for Komodo health'); event: { kind: 'resumed', message: 'Resumed verification of the existing server' } });
return true; return true;
})(); })();
} }
export function finishDeployment(id: string, result: { ok: boolean; message?: string; warnings?: string[] }): void { export function finishDeployment(id: string, result: { ok: boolean; message?: string; warnings?: string[] }): void {
const state = !result.ok ? 'failed' : result.warnings?.length ? 'warning' : 'done'; const state = !result.ok ? 'failed' : result.warnings?.length ? 'warning' : 'done';
db().transaction(() => {
const finishedAt = new Date().toISOString();
db().prepare(`UPDATE server_deployments SET state = ?, finished_at = ?, error = ?, warnings = ? WHERE id = ?`) db().prepare(`UPDATE server_deployments SET state = ?, finished_at = ?, error = ?, warnings = ? WHERE id = ?`)
.run(state, new Date().toISOString(), result.message ?? null, JSON.stringify(result.warnings ?? []), id); .run(state, finishedAt, result.message ?? null, JSON.stringify(result.warnings ?? []), id);
for (const warning of result.warnings ?? []) recordDeploymentEvent(id, warning); for (const warning of result.warnings ?? []) recordProgress(id, { event: { kind: 'warning', message: warning } });
recordDeploymentEvent(id, result.ok ? (state === 'warning' ? 'Completed with warnings' : 'Completed') : `Failed: ${result.message ?? 'Unknown error'}`); recordProgress(id, { blocker: null, event: { kind: 'outcome',
message: result.ok ? (state === 'warning' ? 'Completed with warnings' : 'Completed') : `Failed: ${result.message ?? 'Unknown error'}`,
details: { state, finishedAt, error: result.message ?? null, warnings: result.warnings ?? [] } } });
})();
}
/** Legacy failures also need a durable structured result before same-ID resume clears it. */
function preserveOutcome(id: string): void {
const row = db().prepare('SELECT state, finished_at, error, warnings, observation_json FROM server_deployments WHERE id = ?').get(id) as
{ state: string; finished_at: string | null; error: string | null; warnings: string; observation_json: string | null } | undefined;
if (!row || !['failed', 'interrupted', 'warning', 'done', 'cancelled'].includes(row.state)) return;
const previous = db().prepare("SELECT details_json FROM server_deployment_events WHERE deployment_id = ? AND kind = 'outcome' ORDER BY id DESC LIMIT 1")
.get(id) as { details_json: string | null } | undefined;
try {
const details = JSON.parse(previous?.details_json ?? 'null');
if (details?.state === row.state && details?.finishedAt === row.finished_at) return;
} catch { /* Preserve the row's known outcome when old event metadata is missing. */ }
recordProgress(id, { observedAt: readProgress(row)?.observedAt ?? null, event: { kind: 'outcome',
message: `${row.state === 'interrupted' ? 'Interrupted' : 'Recorded outcome'}: ${row.error ?? row.state}`,
details: { state: row.state, finishedAt: row.finished_at, error: row.error, warnings: warningMessages(row.warnings) } } });
} }
export function listDeployments() { export function listDeployments() {
@@ -132,9 +242,12 @@ export function listDeployments() {
} }
export function interruptDeployments(): void { export function interruptDeployments(): void {
db().transaction(() => {
const running = db().prepare("SELECT id FROM server_deployments WHERE state = 'running'").all() as { id: string }[]; const running = db().prepare("SELECT id FROM server_deployments WHERE state = 'running'").all() as { id: string }[];
db().prepare(`UPDATE server_deployments SET state = 'interrupted', finished_at = ?, const finishedAt = new Date().toISOString();
error = 'The application restarted before this operation finished. Check the server before retrying.' const error = 'The application restarted before this operation finished. Check the server before retrying.';
WHERE state = 'running'`).run(new Date().toISOString()); db().prepare(`UPDATE server_deployments SET state = 'interrupted', finished_at = ?, error = ? WHERE state = 'running'`).run(finishedAt, error);
for (const { id } of running) recordDeploymentEvent(id, 'Interrupted by application restart. Check the server before retrying.'); for (const { id } of running) recordProgress(id, { blocker: null, event: { kind: 'outcome',
message: 'Interrupted by application restart. Check the server before retrying.', details: { state: 'interrupted', finishedAt, error } } });
})();
} }
+1
View File
@@ -4,4 +4,5 @@ export { requestLifecycle, requestRestart, requestWatchdogLifecycle, escalateWat
export type { LifecycleOperation, RestartSource } from './requests.js'; export type { LifecycleOperation, RestartSource } from './requests.js';
export { queueCreateServer, destroyServer, previewCreation, creationQueue, nextAvailablePort, nextLobbyNumber } from './instances.js'; export { queueCreateServer, destroyServer, previewCreation, creationQueue, nextAvailablePort, nextLobbyNumber } from './instances.js';
export { getDeployment, listDeployments, resumeVerification, resumeCleanup, resumeRebuild, cancelRemoval } from './history.js'; export { getDeployment, listDeployments, resumeVerification, resumeCleanup, resumeRebuild, cancelRemoval } from './history.js';
export { deploymentActivity } from './activity.js';
export { restartSettling, restartRecoveryPending } from './restarts.js'; export { restartSettling, restartRecoveryPending } from './restarts.js';
+178 -48
View File
@@ -1,5 +1,5 @@
import { fleetSettings } from '../settings.js'; import { fleetSettings } from '../settings.js';
import { probePlayerCount, probeServerStatus } from '../fleet/status-evidence.js'; import { probePlayerEvidence, probeServerStatus } from '../fleet/status-evidence.js';
import { beginDeployment, deploymentStage, finishDeployment, recordDeploymentEvent, REMOVING_SERVER, resumeCleanup, resumeVerification, claimRemoval } from './history.js'; import { beginDeployment, deploymentStage, finishDeployment, recordDeploymentEvent, REMOVING_SERVER, resumeCleanup, resumeVerification, claimRemoval } from './history.js';
import { randomUUID } from 'node:crypto'; import { randomUUID } from 'node:crypto';
import type { LaunchPreview } from '$lib/deployment-plan.js'; import type { LaunchPreview } from '$lib/deployment-plan.js';
@@ -25,6 +25,8 @@ import {
import { MAP_CONFIG_FILE } from '../fleet/server-config.js'; import { MAP_CONFIG_FILE } from '../fleet/server-config.js';
import { recordScript } from '../fleet/inventory.js'; import { recordScript } from '../fleet/inventory.js';
import { getVersion } from '../fleet/scripts.js'; import { getVersion } from '../fleet/scripts.js';
import type { ProvisioningBlocker } from '../../provisioning-detail.js';
import { recordProgress, type ProgressUpdate } from './progress.js';
export type CreateServerFailure = export type CreateServerFailure =
| 'bad-request' | 'bad-request'
@@ -147,6 +149,13 @@ function buildSpec(args: {
} }
let creating = false; let creating = false;
async function observeCommand<T>(operation: string, command: () => Promise<T>): Promise<T> {
try { return await command(); }
catch (err) {
recordProgress(operation, { command: { state: 'uncertain', message: 'No confirmed command result was received: ' + (err as Error).message } });
throw err;
}
}
interface PreparedCreation { interface PreparedCreation {
ok: true; name: string; port: number; plutoKey: string; mapEntry: ZombieMapEntry; ok: true; name: string; port: number; plutoKey: string; mapEntry: ZombieMapEntry;
} }
@@ -225,7 +234,8 @@ export function queueCreateServer(input: CreateServerInput, releasingServers: st
db().prepare("UPDATE server_deployments SET state = 'queued' WHERE id = ?").run(id); db().prepare("UPDATE server_deployments SET state = 'queued' WHERE id = ?").run(id);
db().prepare('INSERT INTO server_creation_jobs (deployment_id, server, port, game_mode, map_slug, expected_map) VALUES (?, ?, ?, ?, ?, ?)') db().prepare('INSERT INTO server_creation_jobs (deployment_id, server, port, game_mode, map_slug, expected_map) VALUES (?, ?, ?, ?, ?, ?)')
.run(id, prepared.name, prepared.port, input.gameMode, prepared.mapEntry.slug, prepared.mapEntry.canonical); .run(id, prepared.name, prepared.port, input.gameMode, prepared.mapEntry.slug, prepared.mapEntry.canonical);
deploymentStage(id, 'Queued for server creation'); deploymentStage(id, 'Queued for server creation', { phase: 'startup-queue', recipeStatus: 'queued', observedAt: null,
blocker: { code: 'startup-queue', message: 'Waiting for startup admission and available capacity.', automatic: true } });
return { ok: true, name: prepared.name, port: prepared.port, deploymentId: id }; return { ok: true, name: prepared.name, port: prepared.port, deploymentId: id };
})(); })();
} }
@@ -252,7 +262,8 @@ export function queueRebuildServer(name: string, triggeredBy = 'ui'): CreateServ
.run(id, name, server.game_port, 'zombie', map.slug, map.canonical); .run(id, name, server.game_port, 'zombie', map.slug, map.canonical);
db().prepare('INSERT INTO server_rebuild_jobs (deployment_id) VALUES (?)').run(id); db().prepare('INSERT INTO server_rebuild_jobs (deployment_id) VALUES (?)').run(id);
db().prepare('INSERT INTO server_launch_recipes (deployment_id, recipe) VALUES (?, ?)').run(id, JSON.stringify(recipe)); db().prepare('INSERT INTO server_launch_recipes (deployment_id, recipe) VALUES (?, ?)').run(id, JSON.stringify(recipe));
deploymentStage(id, 'Waiting to rebuild the empty server'); deploymentStage(id, 'Waiting to rebuild the empty server', { phase: 'eligibility', recipeStatus: 'queued', observedAt: null,
blocker: { code: 'eligibility', message: 'Waiting for safe rebuild eligibility checks.', automatic: true } });
return { ok: true as const, name, port: server.game_port!, deploymentId: id }; return { ok: true as const, name, port: server.game_port!, deploymentId: id };
})(); })();
} }
@@ -266,12 +277,15 @@ export async function rebuildServer(id: string): Promise<void> {
if (blocker) { if (blocker) {
// Keep the cleanup checkpoint intact when resuming after remote deletion. // Keep the cleanup checkpoint intact when resuming after remote deletion.
if (job.stage === REMOVING_SERVER) finishDeployment(id, { ok: false, message: blocker }); if (job.stage === REMOVING_SERVER) finishDeployment(id, { ok: false, message: blocker });
else deploymentStage(id, blocker); else deploymentStage(id, blocker, { phase: 'eligibility', blocker: betaRemovalEvidence(job.server, blocker), checks: [] });
return; return;
} }
if (creating || destroying.size || creationQueue().active.length >= fleetSettings().createConcurrency) { if (creating || destroying.size || creationQueue().active.length >= fleetSettings().createConcurrency) {
// The cleanup checkpoint must survive capacity waits and app restarts. // The cleanup checkpoint must survive capacity waits and app restarts.
if (job.stage !== REMOVING_SERVER) deploymentStage(id, 'Waiting for a server creation slot before rebuilding'); const progress: ProgressUpdate = { phase: job.stage === REMOVING_SERVER ? 'cleanup' : 'eligibility',
blocker: { code: 'startup-capacity', message: 'Waiting for a server creation slot before rebuilding', automatic: true } };
if (job.stage !== REMOVING_SERVER) deploymentStage(id, 'Waiting for a server creation slot before rebuilding', progress);
else recordProgress(id, progress);
return; return;
} }
// Recheck prerequisites while the original instance still exists. // Recheck prerequisites while the original instance still exists.
@@ -290,7 +304,9 @@ export async function rebuildServer(id: string): Promise<void> {
db().transaction(() => { db().transaction(() => {
db().prepare('UPDATE server_rebuild_jobs SET removed_at = ? WHERE deployment_id = ?').run(new Date().toISOString(), id); db().prepare('UPDATE server_rebuild_jobs SET removed_at = ? WHERE deployment_id = ?').run(new Date().toISOString(), id);
db().prepare("UPDATE server_deployments SET state = 'queued' WHERE id = ?").run(id); db().prepare("UPDATE server_deployments SET state = 'queued' WHERE id = ?").run(id);
deploymentStage(id, 'Queued for server creation'); deploymentStage(id, 'Queued for server creation', { phase: 'startup-queue', checks: [],
blocker: { code: 'startup-queue', message: 'Old instance removed; waiting for fresh-instance startup admission.', automatic: true },
checkpoint: { key: 'cleanup', label: 'Old instance files and registry removed' } });
})(); })();
} catch (err) { finishDeployment(id, { ok: false, message: (err as Error).message }); } } catch (err) { finishDeployment(id, { ok: false, message: (err as Error).message }); }
finally { destroying.delete(job.server); } finally { destroying.delete(job.server); }
@@ -369,11 +385,15 @@ async function createServerUnlocked(input: CreateServerInput, prepared: Prepared
// Resolve once more at provisioning so promotions made while queued also apply. // Resolve once more at provisioning so promotions made while queued also apply.
// Persist the exact attempted files for history, never as input to future launches. // Persist the exact attempted files for history, never as input to future launches.
const recipe = captureRecipe(game, mapEntry.slug); const recipe = captureRecipe(game, mapEntry.slug);
db().transaction(() => {
db().prepare(`INSERT INTO server_launch_recipes (deployment_id, recipe) VALUES (?, ?) db().prepare(`INSERT INTO server_launch_recipes (deployment_id, recipe) VALUES (?, ?)
ON CONFLICT(deployment_id) DO UPDATE SET recipe = excluded.recipe`) ON CONFLICT(deployment_id) DO UPDATE SET recipe = excluded.recipe`)
.run(operation, JSON.stringify(recipe)); .run(operation, JSON.stringify(recipe));
deploymentStage(operation, 'Preparing server files', { phase: 'prepare', recipeStatus: 'attempted', blocker: null, checks: [],
checkpoint: { key: 'startup-queue', label: 'Admitted to startup' },
event: { kind: 'recipe', message: 'Promoted files selected for this creation attempt.', details: { capturedAt: recipe.capturedAt } } });
})();
const configFile = recipe.configFile; const configFile = recipe.configFile;
deploymentStage(operation, 'Preparing server files');
const hostDir = serverDir(name); const hostDir = serverDir(name);
// Step 1: filesystem. mkdir mode is masked by umask, so chmod explicitly // Step 1: filesystem. mkdir mode is masked by umask, so chmod explicitly
@@ -398,13 +418,18 @@ async function createServerUnlocked(input: CreateServerInput, prepared: Prepared
} }
// Step 2: Komodo CreateDeployment. // Step 2: Komodo CreateDeployment.
deploymentStage(operation, 'Creating Komodo deployment'); deploymentStage(operation, 'Creating Komodo deployment', { phase: 'create-deployment',
checkpoint: { key: 'prepare', label: 'Server files prepared' },
command: { state: 'attempted', message: 'Komodo deployment creation attempted.' } });
const spec = buildSpec({ game, name, port, hostDir, plutoKey, recipe }); const spec = buildSpec({ game, name, port, hostDir, plutoKey, recipe });
const createRes = await komodoCreateDeployment(spec); const createRes = await observeCommand(operation, () => komodoCreateDeployment(spec));
if (!createRes.ok) { if (!createRes.ok) {
recordProgress(operation, { command: { state: createRes.kind === 'unreachable' || (createRes.httpStatus !== undefined && createRes.httpStatus >= 500) ? 'uncertain' : 'rejected', message: createRes.message } });
await rm(hostDir, { recursive: true, force: true }).catch(() => {}); await rm(hostDir, { recursive: true, force: true }).catch(() => {});
return { ok: false, kind: 'komodo', message: `CreateDeployment: ${createRes.message}` }; return { ok: false, kind: 'komodo', message: `CreateDeployment: ${createRes.message}` };
} }
recordProgress(operation, { command: { state: 'accepted', message: 'Komodo created the deployment.' },
checkpoint: { key: 'remote-deployment-created', phase: 'create-deployment', label: 'Komodo deployment created' } });
// Step 3: DB row — insert *before* execute/Deploy so the sweep sees the // Step 3: DB row — insert *before* execute/Deploy so the sweep sees the
// server immediately even if Deploy hangs. // server immediately even if Deploy hangs.
@@ -446,23 +471,31 @@ async function createServerUnlocked(input: CreateServerInput, prepared: Prepared
void applyDeploymentTags(name, game, mapEntry.canonical); void applyDeploymentTags(name, game, mapEntry.canonical);
// Step 5: execute/Deploy — actually start the container. // Step 5: execute/Deploy — actually start the container.
deploymentStage(operation, 'Starting container'); db().transaction(() => {
deploymentStage(operation, 'Starting container', { phase: 'container-action',
checkpoint: { key: 'create-deployment', label: 'Deployment created and server registered' },
command: { state: 'attempted', message: 'Container startup submission attempted.' } });
db().prepare('UPDATE server_deployments SET submitted_at = ? WHERE id = ?').run(new Date().toISOString(), operation); db().prepare('UPDATE server_deployments SET submitted_at = ? WHERE id = ?').run(new Date().toISOString(), operation);
const deployRes = await komodoDeploy(name); })();
const deployRes = await observeCommand(operation, () => komodoDeploy(name));
if (!deployRes.ok) { if (!deployRes.ok) {
if (deployRes.kind === 'unreachable' || (deployRes.httpStatus !== undefined && deployRes.httpStatus >= 500)) { if (deployRes.kind === 'unreachable' || (deployRes.httpStatus !== undefined && deployRes.httpStatus >= 500)) {
// A timeout or server error may arrive after the remote boot began. // A timeout or server error may arrive after the remote boot began.
// Preserve files, identity, and capacity until readiness is known. // Preserve files, identity, and capacity until readiness is known.
recordDeploymentEvent(operation, 'Startup response uncertain; verifying the existing server: ' + deployRes.message); recordDeploymentEvent(operation, 'Startup response uncertain; verifying the existing server: ' + deployRes.message);
recordProgress(operation, { command: { state: 'uncertain', message: 'Startup may have begun: ' + deployRes.message } });
return { ok: true, name, port, deploymentId: operation }; return { ok: true, name, port, deploymentId: operation };
} }
// Container never started — undo everything so retrying is clean. // Container never started — undo everything so retrying is clean.
recordProgress(operation, { command: { state: 'rejected', message: 'Startup request rejected: ' + deployRes.message } });
await komodoDeleteDeployment(name).catch(() => {}); await komodoDeleteDeployment(name).catch(() => {});
db().prepare('DELETE FROM servers WHERE name = ?').run(name); db().prepare('DELETE FROM servers WHERE name = ?').run(name);
await rm(hostDir, { recursive: true, force: true }).catch(() => {}); await rm(hostDir, { recursive: true, force: true }).catch(() => {});
return { ok: false, kind: 'komodo', message: `Deploy: ${deployRes.message}` }; return { ok: false, kind: 'komodo', message: `Deploy: ${deployRes.message}` };
} }
recordProgress(operation, { command: { state: 'accepted', message: 'Komodo accepted startup; readiness still requires verification.' },
checkpoint: { key: 'container-action', label: 'Container startup accepted' } });
return { ok: true, name, port, deploymentId: operation }; return { ok: true, name, port, deploymentId: operation };
} }
@@ -515,7 +548,8 @@ export function destroyServer(name: string, triggeredBy = 'ui'): DestroyServerRe
const operation = db().transaction(() => { const operation = db().transaction(() => {
const id = beginDeployment(name, 'destroy', triggeredBy); const id = beginDeployment(name, 'destroy', triggeredBy);
db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id); db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id);
deploymentStage(id, 'Waiting to verify the server is empty'); deploymentStage(id, 'Waiting to verify the server is empty', { phase: 'eligibility', observedAt: null,
blocker: { code: 'eligibility', message: 'Waiting for safe removal eligibility checks.', automatic: true } });
return id; return id;
})(); })();
return { ok: true, warnings: [], deploymentId: operation }; return { ok: true, warnings: [], deploymentId: operation };
@@ -528,6 +562,13 @@ export function processDestructions(): Promise<void> {
return destructionTick; return destructionTick;
} }
function betaRemovalEvidence(server: string, message: string): ProvisioningBlocker {
const beta = db().prepare(`SELECT b.id FROM beta_servers bs JOIN betas b ON b.id = bs.beta_id
WHERE bs.server = ? AND bs.server_deleted_at IS NULL AND b.state IN ('running','ready-promote') ORDER BY b.id LIMIT 1`)
.get(server) as { id: number } | undefined;
return { code: 'active-beta', message, automatic: true, href: beta ? `/betas/${beta.id}` : null, label: 'Blocking beta' };
}
async function runDestructions(): Promise<void> { async function runDestructions(): Promise<void> {
const queued = db().prepare(`SELECT id, server, stage, server_deleted_at FROM server_deployments WHERE action = 'destroy' AND state = 'waiting' const queued = db().prepare(`SELECT id, server, stage, server_deleted_at FROM server_deployments WHERE action = 'destroy' AND state = 'waiting'
AND (server_deleted_at IS NULL OR stage = ?) ORDER BY started_at`).all(REMOVING_SERVER) as { id: string; server: string; stage: string; server_deleted_at: string | null }[]; AND (server_deleted_at IS NULL OR stage = ?) ORDER BY started_at`).all(REMOVING_SERVER) as { id: string; server: string; stage: string; server_deleted_at: string | null }[];
@@ -553,37 +594,53 @@ async function runDestructions(): Promise<void> {
} }
if (!getServer(operation.server)) { if (!getServer(operation.server)) {
if (hasActiveCreation(operation.server)) { if (hasActiveCreation(operation.server)) {
deploymentStage(operation.id, 'Waiting for server creation to finish'); const creatingOperation = db().prepare(`SELECT id FROM server_deployments WHERE server = ? AND server_deleted_at IS NULL
AND action IN ('create','rebuild') AND state IN ('queued','running','awaiting-health') LIMIT 1`).get(operation.server) as { id: string } | undefined;
deploymentStage(operation.id, 'Waiting for server creation to finish', { phase: 'eligibility', checks: [],
blocker: { code: 'creation-pending', message: 'Waiting for server creation to finish', automatic: true,
href: creatingOperation ? `/provisioning/${encodeURIComponent(creatingOperation.id)}` : null, label: 'Creation operation' } });
continue; continue;
} }
finishDeployment(operation.id, { ok: false, message: 'Server is no longer registered' }); finishDeployment(operation.id, { ok: false, message: 'Server is no longer registered' });
continue; continue;
} }
const blocker = activeBetaRemovalBlocker(operation.server); const blocker = activeBetaRemovalBlocker(operation.server);
if (blocker) { deploymentStage(operation.id, blocker); continue; } if (blocker) { deploymentStage(operation.id, blocker, { phase: 'eligibility', checks: [], blocker: betaRemovalEvidence(operation.server, blocker) }); continue; }
// A fresh successful zero is required; cached occupancy and timeouts are not evidence of emptiness. // A fresh successful zero is required; cached occupancy and timeouts are not evidence of emptiness.
let players = await probePlayerCount(operation.server); const evidence = await probePlayerEvidence(operation.server);
let players = evidence.players;
let occupancy: NonNullable<ProgressUpdate['checks']>[number] = { key: 'occupancy', label: 'Player occupancy',
result: players === 0 ? 'pass' : players === null ? 'unknown' : 'waiting', source: evidence.source,
checkedAt: evidence.observedAt, receivedAt: evidence.receivedAt, expected: 0, actual: players };
if (players === null) { if (players === null) {
const health = await komodoInspectHealthy(operation.server); const health = await komodoInspectHealthy(operation.server);
if (['stopped','exited','created','dead'].includes(health.state?.toLowerCase() ?? '')) players = 0; if (['stopped','exited','created','dead'].includes(health.state?.toLowerCase() ?? '')) {
players = 0;
occupancy = { key: 'stopped', label: 'Stopped container permits removal', result: 'pass', source: 'komodo', actual: health.state ?? null };
}
} }
if (!db().prepare("SELECT id FROM server_deployments WHERE id = ? AND state = 'waiting'").get(operation.id)) continue; if (!db().prepare("SELECT id FROM server_deployments WHERE id = ? AND state = 'waiting'").get(operation.id)) continue;
// Beta ownership may have changed while the network probes were in flight. // Beta ownership may have changed while the network probes were in flight.
const latestBlocker = activeBetaRemovalBlocker(operation.server); const latestBlocker = activeBetaRemovalBlocker(operation.server);
if (latestBlocker) { deploymentStage(operation.id, latestBlocker); continue; } if (latestBlocker) { deploymentStage(operation.id, latestBlocker, { phase: 'eligibility', checks: [occupancy], blocker: betaRemovalEvidence(operation.server, latestBlocker) }); continue; }
if (players !== 0) { if (players !== 0) {
deploymentStage(operation.id, players === null const message = players === null
? 'Waiting for player count: server could not be reached' ? 'Waiting for player count: server could not be reached'
: 'Waiting for server to empty: ' + players + ' player(s) online'); : 'Waiting for server to empty: ' + players + ' player(s) online';
deploymentStage(operation.id, message, { phase: 'eligibility', checks: [occupancy],
blocker: { code: players === null ? 'players-unknown' : 'players-present', message, automatic: true } });
continue; continue;
} }
if (!claimRemoval(operation.id)) continue; if (!claimRemoval(operation.id)) continue;
deploymentStage(operation.id, 'Server is empty (0 players verified)'); deploymentStage(operation.id, 'Server is empty (0 players verified)', { phase: 'eligibility', checks: [occupancy], blocker: null,
checkpoint: { key: 'eligibility', label: 'Removal eligibility verified' } });
const result = await destroyServerUnlocked(operation.server, operation.id); const result = await destroyServerUnlocked(operation.server, operation.id);
finishDeployment(operation.id, result); finishDeployment(operation.id, result);
} catch (err) { } catch (err) {
const row = db().prepare('SELECT state FROM server_deployments WHERE id = ?').get(operation.id) as { state: string }; const row = db().prepare('SELECT state FROM server_deployments WHERE id = ?').get(operation.id) as { state: string };
if (row.state === 'waiting') deploymentStage(operation.id, 'Waiting for player count: ' + (err as Error).message); if (row.state === 'waiting') deploymentStage(operation.id, 'Waiting for player count: ' + (err as Error).message, {
phase: 'eligibility', checks: [], blocker: { code: 'players-unknown', message: (err as Error).message, automatic: true }
});
else if (row.state !== 'cancelled') finishDeployment(operation.id, { ok: false, message: (err as Error).message }); else if (row.state !== 'cancelled') finishDeployment(operation.id, { ok: false, message: (err as Error).message });
} finally { destroying.delete(operation.server); } } finally { destroying.delete(operation.server); }
} }
@@ -601,11 +658,15 @@ async function destroyServerUnlocked(name: string, operation: string): Promise<D
// 1: stop + remove the container. execute/DestroyDeployment queues the // 1: stop + remove the container. execute/DestroyDeployment queues the
// teardown; the response comes back before the container is actually gone. // teardown; the response comes back before the container is actually gone.
deploymentStage(operation, 'Stopping and removing container'); deploymentStage(operation, 'Stopping and removing container', { phase: 'remove-container', blocker: null, checks: [],
const destroyRes = await komodoDestroyDeployment(name); checkpoint: { key: 'eligibility', label: 'Removal eligibility verified' },
command: { state: 'attempted', message: 'Container removal submission attempted.' } });
const destroyRes = await observeCommand(operation, () => komodoDestroyDeployment(name));
if (!destroyRes.ok) { if (!destroyRes.ok) {
recordProgress(operation, { command: { state: destroyRes.kind === 'unreachable' || (destroyRes.httpStatus !== undefined && destroyRes.httpStatus >= 500) ? 'uncertain' : 'rejected', message: destroyRes.message } });
return { ok: false, kind: 'komodo', message: 'DestroyDeployment failed: ' + destroyRes.message, warnings }; return { ok: false, kind: 'komodo', message: 'DestroyDeployment failed: ' + destroyRes.message, warnings };
} }
recordProgress(operation, { command: { state: 'accepted', message: 'Komodo accepted teardown; removal is not yet confirmed.' } });
// 2: wait for the container to actually disappear. Komodo refuses to // 2: wait for the container to actually disappear. Komodo refuses to
// DeleteDeployment while a container still exists for it, so poll the // DeleteDeployment while a container still exists for it, so poll the
@@ -619,12 +680,19 @@ async function destroyServerUnlocked(name: string, operation: string): Promise<D
} }
// 3: remove the deployment entity itself. // 3: remove the deployment entity itself.
deploymentStage(operation, 'Deleting Komodo deployment'); deploymentStage(operation, 'Deleting Komodo deployment', { phase: 'delete-deployment',
const deleteRes = await komodoDeleteDeployment(name); command: { state: 'attempted', message: 'Komodo deployment deletion attempted.' } });
const deleteRes = await observeCommand(operation, () => komodoDeleteDeployment(name));
if (!deleteRes.ok) { if (!deleteRes.ok) {
recordProgress(operation, { command: { state: deleteRes.kind === 'unreachable' || (deleteRes.httpStatus !== undefined && deleteRes.httpStatus >= 500) ? 'uncertain' : 'rejected', message: deleteRes.message } });
return { ok: false, kind: 'komodo', message: 'DeleteDeployment failed: ' + deleteRes.message, warnings }; return { ok: false, kind: 'komodo', message: 'DeleteDeployment failed: ' + deleteRes.message, warnings };
} }
deploymentStage(operation, REMOVING_SERVER, { phase: 'cleanup', blocker: null, checks: [],
command: { state: 'accepted', message: 'Komodo confirmed deployment deletion.' },
checkpoints: [{ key: 'remove-container', label: 'Container removal confirmed by deployment deletion' },
{ key: 'delete-deployment', label: 'Komodo deployment deletion confirmed' }] });
return cleanupDestroyedServer(name, operation); return cleanupDestroyedServer(name, operation);
} }
@@ -635,12 +703,14 @@ async function cleanupDestroyedServer(name: string, operation: string): Promise<
const warnings: string[] = []; const warnings: string[] = [];
// Persist the checkpoint before local work so failed cleanup can resume without RCON. // Persist the checkpoint before local work so failed cleanup can resume without RCON.
// Legacy operations already recorded this same stage after Komodo deletion. // Legacy operations already recorded this same stage after Komodo deletion.
deploymentStage(operation, REMOVING_SERVER); deploymentStage(operation, REMOVING_SERVER, { phase: 'cleanup', blocker: null, checks: [],
checkpoint: { key: 'delete-deployment', label: 'Komodo deployment deletion confirmed' } });
// Host directory. Fall back to derived path if host_dir wasn't set on // Host directory. Fall back to derived path if host_dir wasn't set on
// legacy rows. // legacy rows.
const hostDir = row?.host_dir ?? serverDir(name); const hostDir = row?.host_dir ?? serverDir(name);
try { try {
await rm(hostDir, { recursive: true, force: true }); await rm(hostDir, { recursive: true, force: true });
recordProgress(operation, { checkpoint: { key: 'files-removed', label: 'Instance files removed', phase: 'cleanup' } });
} catch (err) { } catch (err) {
warnings.push(`rm ${hostDir} failed: ${(err as Error).message}`); warnings.push(`rm ${hostDir} failed: ${(err as Error).message}`);
} }
@@ -648,7 +718,14 @@ async function cleanupDestroyedServer(name: string, operation: string): Promise<
// The registry trigger preserves history and marks its old server identity. // The registry trigger preserves history and marks its old server identity.
// Live inventory, rollback snapshots, and restart slots still cascade away. // Live inventory, rollback snapshots, and restart slots still cascade away.
try { try {
db().transaction(() => {
const recovery = db().prepare('SELECT operation_id FROM restart_rollout WHERE server = ? AND healthy_at IS NULL').get(name) as { operation_id: string | null } | undefined;
db().prepare('DELETE FROM servers WHERE name = ?').run(name); db().prepare('DELETE FROM servers WHERE name = ?').run(name);
if (recovery?.operation_id) recordProgress(recovery.operation_id, { recovery: { state: 'released', operationId: recovery.operation_id,
message: 'Recovery capacity released when the old instance registry was removed.' } });
recordProgress(operation, { checkpoint: { key: 'registry-removed', label: 'Server registration removed', phase: 'cleanup' } });
if (!warnings.length) recordProgress(operation, { checkpoint: { key: 'cleanup', label: 'Local files and registry cleanup completed' } });
})();
} catch (err) { } catch (err) {
return { return {
ok: false, ok: false,
@@ -684,8 +761,11 @@ async function provisionCreation(id: string): Promise<CreateServerResult> {
{ ok: true, name: job.server, port: job.port, mapEntry, plutoKey }, id); { ok: true, name: job.server, port: job.port, mapEntry, plutoKey }, id);
if (!result.ok) finishDeployment(id, result); if (!result.ok) finishDeployment(id, result);
else { else {
db().transaction(() => {
db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ?").run(id); db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ?").run(id);
deploymentStage(id, 'Waiting for Komodo health'); deploymentStage(id, 'Waiting for Komodo health', { phase: 'verify-container', checks: [],
blocker: { code: 'container-readiness', message: 'Waiting for container readiness.', automatic: true } });
})();
} }
return result; return result;
} catch (err) { } catch (err) {
@@ -693,9 +773,13 @@ async function provisionCreation(id: string): Promise<CreateServerResult> {
if (server?.game_port === job.port) { if (server?.game_port === job.port) {
// A lost response may have happened after Komodo started the server. // A lost response may have happened after Komodo started the server.
// Keep its slot and observe readiness instead of starting another boot. // Keep its slot and observe readiness instead of starting another boot.
db().transaction(() => {
db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ?").run(id); db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ?").run(id);
recordDeploymentEvent(id, 'Startup response interrupted: ' + (err as Error).message); recordDeploymentEvent(id, 'Startup response interrupted: ' + (err as Error).message);
deploymentStage(id, 'Verifying existing server after interrupted startup'); deploymentStage(id, 'Verifying existing server after interrupted startup', { phase: 'verify-container', checks: [],
command: { state: 'uncertain', message: 'Startup response interrupted: ' + (err as Error).message },
blocker: { code: 'container-readiness', message: 'Observing the existing instance after an uncertain startup.', automatic: true } });
})();
} else finishDeployment(id, { ok: false, message: (err as Error).message }); } else finishDeployment(id, { ok: false, message: (err as Error).message });
throw err; throw err;
} finally { creating = false; } } finally { creating = false; }
@@ -722,19 +806,32 @@ export function creationQueue() {
return { active: jobs.filter(job => job.state !== 'queued'), queued: jobs.filter(job => job.state === 'queued') }; return { active: jobs.filter(job => job.state !== 'queued'), queued: jobs.filter(job => job.state === 'queued') };
} }
function creationBlocker(id: string): string | null { function creationBlocker(id: string): ProvisioningBlocker | null {
const dependency = db().prepare(`SELECT d.server, d.state FROM server_creation_dependencies dep const dependency = db().prepare(`SELECT d.id, d.server, d.state FROM server_creation_dependencies dep
JOIN server_deployments d ON d.id = dep.removal_id JOIN server_deployments d ON d.id = dep.removal_id
WHERE dep.deployment_id = ? AND d.state NOT IN ('done', 'warning') LIMIT 1`).get(id) as { server: string; state: string } | undefined; WHERE dep.deployment_id = ? AND d.state NOT IN ('done', 'warning') LIMIT 1`).get(id) as { id: string; server: string; state: string } | undefined;
if (dependency) return `Waiting for removal of ${dependency.server} (${dependency.state}) before reusing its port`; if (dependency) return { code: 'removal-dependency', message: `Waiting for removal of ${dependency.server} (${dependency.state}) before reusing its port`,
automatic: true, href: `/provisioning/${encodeURIComponent(dependency.id)}`, label: 'Required removal' };
const occupied = db().prepare(`SELECT s.name FROM servers s JOIN server_creation_jobs j ON s.game_port = j.port const occupied = db().prepare(`SELECT s.name FROM servers s JOIN server_creation_jobs j ON s.game_port = j.port
WHERE j.deployment_id = ? LIMIT 1`).get(id) as { name: string } | undefined; WHERE j.deployment_id = ? LIMIT 1`).get(id) as { name: string } | undefined;
return occupied ? `Waiting for port to be released by ${occupied.name}` : null; return occupied ? { code: 'port-occupied', message: `Waiting for port to be released by ${occupied.name}`, automatic: true,
href: `/servers/${encodeURIComponent(occupied.name)}`, label: 'Server using this port' } : null;
}
function stillCheckingCreation(id: string): boolean {
return !!db().prepare(`SELECT 1 FROM server_deployments d JOIN server_creation_jobs j ON j.deployment_id = d.id
JOIN servers s ON s.name = j.server AND s.game_port = j.port
WHERE d.id = ? AND d.server_deleted_at IS NULL AND d.state = 'awaiting-health'`).get(id);
} }
async function runCreations(): Promise<CreationOutcomes> { async function runCreations(): Promise<CreationOutcomes> {
const outcomes: CreationOutcomes = new Map(); const outcomes: CreationOutcomes = new Map();
if (creating || destroying.size > 0) return outcomes; if (creating || destroying.size > 0) {
for (const job of creationQueue().queued) recordProgress(job.id, { phase: 'startup-queue', blocker: {
code: 'startup-worker-busy', message: 'Waiting for in-flight instance creation or removal to finish.', automatic: true
} });
return outcomes;
}
// Restart recovery is observation-only: never repeat external creation or // Restart recovery is observation-only: never repeat external creation or
// deployment. A known interrupted startup must retain its readiness gate. // deployment. A known interrupted startup must retain its readiness gate.
const interrupted = db().prepare("SELECT id FROM server_deployments WHERE server_deleted_at IS NULL AND action IN ('create', 'rebuild') AND state = 'interrupted' ORDER BY started_at, rowid").all() as { id: string }[]; const interrupted = db().prepare("SELECT id FROM server_deployments WHERE server_deleted_at IS NULL AND action IN ('create', 'rebuild') AND state = 'interrupted' ORDER BY started_at, rowid").all() as { id: string }[];
@@ -752,7 +849,8 @@ async function runCreations(): Promise<CreationOutcomes> {
AND newer.rowid > d.rowid AND newer.action = 'create')`).get(id); AND newer.rowid > d.rowid AND newer.action = 'create')`).get(id);
if (removing) { if (removing) {
db().prepare("UPDATE server_deployments SET state = 'awaiting-health', finished_at = NULL, error = NULL WHERE id = ?").run(id); db().prepare("UPDATE server_deployments SET state = 'awaiting-health', finished_at = NULL, error = NULL WHERE id = ?").run(id);
deploymentStage(id, 'Waiting for requested server removal after interrupted startup'); deploymentStage(id, 'Waiting for requested server removal after interrupted startup', { checks: [],
blocker: { code: 'removal-pending', message: 'Waiting for the requested removal; the interrupted startup still occupies capacity.', automatic: true } });
} }
} }
@@ -766,21 +864,51 @@ async function runCreations(): Promise<CreationOutcomes> {
finishDeployment(id, { ok: false, message: 'Server was removed or replaced before creation health checks completed' }); finishDeployment(id, { ok: false, message: 'Server was removed or replaced before creation health checks completed' });
continue; continue;
} }
if (db().prepare("SELECT id FROM server_deployments WHERE server = ? AND server_deleted_at IS NULL AND action = 'destroy' AND state IN ('waiting', 'running')").get(job.server)) continue; const removal = db().prepare("SELECT id FROM server_deployments WHERE server = ? AND server_deleted_at IS NULL AND action = 'destroy' AND state IN ('waiting', 'running')").get(job.server) as { id: string } | undefined;
try { if (removal) {
const komodo = await komodoInspectHealthy(job.server); recordProgress(id, { checks: [], blocker: { code: 'removal-pending', message: 'Creation verification is held while server removal is pending.', automatic: true,
if (!komodo.ok) { deploymentStage(id, 'Waiting for Komodo health: ' + komodo.reason); continue; } href: `/provisioning/${encodeURIComponent(removal.id)}`, label: 'Removal operation' } });
const status = await probeServerStatus(job.server, job.submitted_at);
if (!status.ok) { deploymentStage(id, 'Waiting for RCON or fresh telemetry health: ' + status.reason); continue; }
const actualMap = status.map;
if (actualMap !== job.expected_map) {
deploymentStage(id, 'Waiting for selected map: expected ' + job.expected_map + ', ' + (status.source === 'rcon' ? 'RCON' : 'telemetry') + ' reports ' + (actualMap ?? 'no map'));
continue; continue;
} }
deploymentStage(id, 'Verified Komodo healthy, ' + (status.source === 'rcon' ? 'RCON healthy' : 'fresh telemetry') + ', and map ' + actualMap); const checks: NonNullable<ProgressUpdate['checks']> = [];
try {
const komodo = await komodoInspectHealthy(job.server);
if (!stillCheckingCreation(id)) continue;
checks.push({ key: 'container', label: 'Container readiness', source: 'komodo', result: komodo.ok ? 'pass' : 'waiting',
checkedAt: new Date().toISOString(), actual: komodo.state ?? null, message: komodo.ok ? null : komodo.reason });
if (!komodo.ok) { deploymentStage(id, 'Waiting for Komodo health: ' + komodo.reason, { phase: 'verify-container', checks,
blocker: { code: 'container-readiness', message: komodo.reason, automatic: true } }); continue; }
const status = await probeServerStatus(job.server, job.submitted_at);
if (!stillCheckingCreation(id)) continue;
checks.push({ key: 'game', label: 'Game status', source: status.ok ? status.source : null, result: status.ok ? 'pass' : 'waiting',
checkedAt: status.observedAt, receivedAt: status.ok ? status.receivedAt : null, actual: status.ok ? status.players : null, message: status.ok ? null : status.reason });
if (!status.ok) { deploymentStage(id, 'Waiting for RCON or fresh telemetry health: ' + status.reason, { phase: 'verify-game', checks,
blocker: { code: 'game-readiness', message: status.reason, automatic: true } }); continue; }
const actualMap = status.map;
checks.push({ key: 'map', label: 'Selected engine map', source: status.source, result: actualMap === job.expected_map ? 'pass' : 'waiting',
checkedAt: status.observedAt, receivedAt: status.receivedAt, expected: job.expected_map, actual: actualMap });
if (actualMap !== job.expected_map) {
const message = 'Waiting for selected map: expected ' + job.expected_map + ', ' + (status.source === 'rcon' ? 'RCON' : 'telemetry') + ' reports ' + (actualMap ?? 'no map');
deploymentStage(id, message, { phase: 'verify-map', checks, blocker: { code: 'map-mismatch', message, automatic: true,
href: `/servers/${encodeURIComponent(job.server)}?tab=config`, label: 'Map configuration' } });
continue;
}
db().transaction(() => {
deploymentStage(id, 'Verified Komodo healthy, ' + (status.source === 'rcon' ? 'RCON healthy' : 'fresh telemetry') + ', and map ' + actualMap, {
phase: 'verify-map', checks, blocker: null, checkpoints: [
{ key: 'container-action', label: 'Container startup verified' },
{ key: 'verify-container', label: 'Container readiness verified' },
{ key: 'verify-game', label: 'Game status verified' },
{ key: 'verify-map', label: 'Selected engine map verified' }
]
});
finishDeployment(id, { ok: true }); finishDeployment(id, { ok: true });
})();
} catch (err) { } catch (err) {
deploymentStage(id, 'Waiting for health verification: ' + (err as Error).message); if (!stillCheckingCreation(id)) continue;
checks.push({ key: checks.length ? 'game' : 'container', label: checks.length ? 'Game status' : 'Container readiness', source: null, result: 'unknown', message: (err as Error).message });
deploymentStage(id, 'Waiting for health verification: ' + (err as Error).message, { phase: checks.length > 1 ? 'verify-game' : 'verify-container', checks,
blocker: { code: 'observation-unavailable', message: (err as Error).message, automatic: true } });
} }
} }
@@ -790,7 +918,9 @@ async function runCreations(): Promise<CreationOutcomes> {
const available = Math.max(0, fleetSettings().createConcurrency - queue.active.length); const available = Math.max(0, fleetSettings().createConcurrency - queue.active.length);
const eligible = queue.queued.filter(({ id }) => { const eligible = queue.queued.filter(({ id }) => {
const blocker = creationBlocker(id); const blocker = creationBlocker(id);
if (blocker) deploymentStage(id, blocker); if (blocker) deploymentStage(id, blocker.message, { phase: 'startup-queue', blocker });
else recordProgress(id, { phase: 'startup-queue', blocker: { code: available ? 'startup-queue' : 'startup-capacity',
message: available ? 'Eligible for startup admission in request order.' : 'Waiting for startup capacity to be released.', automatic: true } });
return !blocker; return !blocker;
}); });
for (const { id } of eligible.slice(0, available)) { for (const { id } of eligible.slice(0, available)) {
@@ -801,7 +931,7 @@ async function runCreations(): Promise<CreationOutcomes> {
if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND action IN ('create', 'rebuild') AND state = 'queued'").get(id)) continue; if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND action IN ('create', 'rebuild') AND state = 'queued'").get(id)) continue;
const blocker = creationBlocker(id); const blocker = creationBlocker(id);
if (blocker) { if (blocker) {
deploymentStage(id, blocker); deploymentStage(id, blocker.message, { phase: 'startup-queue', blocker });
continue; continue;
} }
try { outcomes.set(id, await provisionCreation(id)); } try { outcomes.set(id, await provisionCreation(id)); }
+120
View File
@@ -0,0 +1,120 @@
import { db } from '../db/index.js';
import { provisioningPhases, type ProvisioningPhase, type ProvisioningProgress, type ProvisioningCheck,
type ProvisioningCheckpoint, type ProvisioningBlocker, type RecoveryObservation } from '../../provisioning-detail.js';
export type CheckInput = Omit<ProvisioningCheck, 'checkedAt' | 'receivedAt' | 'expected' | 'actual' | 'message'> &
Partial<Pick<ProvisioningCheck, 'checkedAt' | 'receivedAt' | 'expected' | 'actual' | 'message'>>;
export type CheckpointInput = Omit<ProvisioningCheckpoint, 'at' | 'phase'> & { at?: string; phase?: ProvisioningPhase | null };
export interface ProgressUpdate {
phase?: ProvisioningPhase | null;
blocker?: ProvisioningBlocker | null;
/** A complete evaluation; omitted checks are no longer current evidence. */
checks?: CheckInput[];
checkpoint?: CheckpointInput;
checkpoints?: CheckpointInput[];
command?: (Omit<NonNullable<ProvisioningProgress['command']>, 'at'> & { at?: string }) | null;
recovery?: (Omit<RecoveryObservation, 'at'> & { at?: string }) | null;
recipeStatus?: ProvisioningProgress['recipeStatus'];
observedAt?: string | null;
event?: { kind: string; message: string; details?: Record<string, unknown> };
}
const object = (value: unknown): value is Record<string, unknown> => !!value && typeof value === 'object' && !Array.isArray(value);
const nullableString = (value: unknown) => value === null || typeof value === 'string';
const phase = (value: unknown): value is ProvisioningPhase => provisioningPhases.includes(value as ProvisioningPhase);
const scalar = (value: unknown) => nullableString(value) || typeof value === 'number' && Number.isFinite(value);
/** Invalid/old snapshots never manufacture evidence. Raw legacy stages remain available. */
export function readProgress(row: { observation_json?: string | null }): ProvisioningProgress | null {
if (!row.observation_json) return null;
try {
const v: unknown = JSON.parse(row.observation_json);
if (!object(v) || v.version !== 1 || !(v.phase === null || phase(v.phase)) || !nullableString(v.phaseStartedAt)
|| !nullableString(v.observedAt) || !Array.isArray(v.checks) || !Array.isArray(v.checkpoints)) return null;
if (v.blocker !== null && (!object(v.blocker) || typeof v.blocker.code !== 'string' || typeof v.blocker.message !== 'string'
|| typeof v.blocker.automatic !== 'boolean' || (v.blocker.href !== undefined && !nullableString(v.blocker.href))
|| (v.blocker.label !== undefined && !nullableString(v.blocker.label)))) return null;
if (v.checks.length > 10 || !v.checks.every(c => object(c) && ['container', 'game', 'map', 'occupancy', 'stopped'].includes(String(c.key))
&& typeof c.label === 'string' && ['pass', 'waiting', 'fail', 'unknown'].includes(String(c.result))
&& (c.source === null || ['komodo', 'rcon', 'telemetry', 'combined'].includes(String(c.source)))
&& typeof c.checkedAt === 'string' && nullableString(c.receivedAt) && scalar(c.expected) && scalar(c.actual) && nullableString(c.message))) return null;
if (v.checkpoints.length > 100 || !v.checkpoints.every(c => object(c) && typeof c.key === 'string' && typeof c.label === 'string'
&& typeof c.at === 'string' && (c.phase === null || phase(c.phase)))) return null;
if (v.command !== null && (!object(v.command) || !['attempted', 'accepted', 'rejected', 'uncertain'].includes(String(v.command.state))
|| typeof v.command.at !== 'string' || typeof v.command.message !== 'string')) return null;
if (v.recovery !== null && (!object(v.recovery) || !['held', 'released', 'transferred', 'unknown'].includes(String(v.recovery.state))
|| typeof v.recovery.at !== 'string' || typeof v.recovery.message !== 'string'
|| (v.recovery.operationId !== undefined && !nullableString(v.recovery.operationId)))) return null;
if (v.recipeStatus !== null && v.recipeStatus !== 'queued' && v.recipeStatus !== 'attempted') return null;
return v as unknown as ProvisioningProgress;
} catch { return null; }
}
/**
* Records facts at the existing execution boundary, never performs a probe or command.
* Can be nested inside the caller's transaction with its state/legacy-stage writes.
*/
export function recordProgress(id: string, update: ProgressUpdate): void {
const handle = db();
handle.transaction(() => {
const row = handle.prepare('SELECT observation_json FROM server_deployments WHERE id = ?').get(id) as
{ observation_json: string | null } | undefined;
if (!row) return;
const now = new Date().toISOString();
const previous = readProgress(row);
const next: ProvisioningProgress = previous ? { ...previous, checkpoints: [...previous.checkpoints] } : {
version: 1, phase: null, phaseStartedAt: null, observedAt: null, blocker: null, checks: [], checkpoints: [],
command: null, recovery: null, recipeStatus: null
};
const events: { kind: string; message: string; details?: Record<string, unknown> }[] = [];
if (update.phase !== undefined && update.phase !== next.phase) {
next.phase = update.phase;
next.phaseStartedAt = update.phase === null ? null : now;
next.blocker = null;
}
next.observedAt = update.observedAt === undefined ? now : update.observedAt;
if (update.blocker !== undefined) {
const old = next.blocker;
next.blocker = update.blocker;
// Counts/error wording can change on every poll without a new kind of hold.
if (update.blocker && (old?.code !== update.blocker.code || old?.href !== update.blocker.href)) {
events.push({ kind: 'blocker', message: update.blocker.message,
details: { code: update.blocker.code, href: update.blocker.href ?? null } });
}
}
if (update.checks !== undefined) {
next.checks = update.checks.map(check => ({ ...check, checkedAt: check.checkedAt ?? now,
receivedAt: check.receivedAt ?? null, expected: check.expected ?? null, actual: check.actual ?? null,
message: check.message ?? null }));
}
for (const input of [...(update.checkpoints ?? []), ...(update.checkpoint ? [update.checkpoint] : [])]) {
if (next.checkpoints.some(c => c.key === input.key)) continue;
const point: ProvisioningCheckpoint = { ...input, at: input.at ?? now, phase: input.phase ?? (phase(input.key) ? input.key : next.phase) };
next.checkpoints.push(point);
events.push({ kind: 'checkpoint', message: point.label, details: { key: point.key, at: point.at } });
}
if (update.command !== undefined) {
if (!update.command) next.command = null;
else if (next.command?.state !== update.command.state || next.command?.message !== update.command.message) {
next.command = { ...update.command, at: update.command.at ?? now };
events.push({ kind: 'command', message: next.command.message, details: { state: next.command.state, at: next.command.at } });
}
}
if (update.recovery !== undefined) {
if (!update.recovery) next.recovery = null;
else {
const changed = next.recovery?.state !== update.recovery.state || next.recovery?.operationId !== update.recovery.operationId;
next.recovery = { ...update.recovery, at: changed ? update.recovery.at ?? now : next.recovery!.at };
if (changed) events.push({ kind: 'recovery', message: next.recovery.message,
details: { state: next.recovery.state, at: next.recovery.at, operationId: next.recovery.operationId ?? null } });
}
}
if (update.recipeStatus !== undefined) next.recipeStatus = update.recipeStatus;
if (update.event) events.push(update.event);
handle.prepare(`UPDATE server_deployments SET phase_key = ?, phase_started_at = ?, observed_at = ?, observation_json = ? WHERE id = ?`)
.run(next.phase, next.phaseStartedAt, next.observedAt, JSON.stringify(next), id);
const insert = handle.prepare(`INSERT INTO server_deployment_events (deployment_id, created_at, message, kind, phase_key, details_json)
VALUES (?, ?, ?, ?, ?, ?)`);
for (const event of events) insert.run(id, now, event.message, event.kind, next.phase, event.details ? JSON.stringify(event.details) : null);
})();
}
+42 -11
View File
@@ -6,6 +6,7 @@ import { beginDeployment, deploymentStage, finishDeployment, recordDeploymentEve
import { queueRebuildServer } from './instances.js'; import { queueRebuildServer } from './instances.js';
import { restartRequestInFlight } from './restarts.js'; import { restartRequestInFlight } from './restarts.js';
import { runtimeActions, type RuntimeAction } from '../../provisioning.js'; import { runtimeActions, type RuntimeAction } from '../../provisioning.js';
import { recordProgress } from './progress.js';
export interface LifecycleOperation { export interface LifecycleOperation {
id: string; server: string; action: string; state: string; stage: string; id: string; server: string; action: string; state: string; stage: string;
@@ -67,7 +68,8 @@ export function requestRestart(server: string, source: RestartSource): Lifecycle
if (!id) { if (!id) {
id = beginDeployment(server, 'restart', `${source.kind}:${source.id}`); id = beginDeployment(server, 'restart', `${source.kind}:${source.id}`);
db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id); db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id);
deploymentStage(id, 'Waiting to verify the server is empty'); deploymentStage(id, 'Waiting to verify the server is empty', { phase: 'eligibility', observedAt: null,
blocker: { code: 'eligibility', message: 'Waiting for eligibility and a fresh empty or stopped server.', automatic: true } });
if (row.restarted_at) adoptSubmittedRestart(id, server, row.restarted_at); if (row.restarted_at) adoptSubmittedRestart(id, server, row.restarted_at);
} }
db().prepare('INSERT OR IGNORE INTO provisioning_sources (operation_id, kind, source_id, server) VALUES (?, ?, ?, ?)') db().prepare('INSERT OR IGNORE INTO provisioning_sources (operation_id, kind, source_id, server) VALUES (?, ?, ?, ?)')
@@ -79,12 +81,17 @@ export function requestRestart(server: string, source: RestartSource): Lifecycle
/** Upgrade observation only: existing restart intent is never resubmitted. */ /** Upgrade observation only: existing restart intent is never resubmitted. */
function adoptSubmittedRestart(id: string, server: string, submittedAt: string): void { function adoptSubmittedRestart(id: string, server: string, submittedAt: string): void {
const slot = db().prepare('SELECT operation_id, requested_at, healthy_at, players FROM restart_rollout WHERE server = ?').get(server) as const slot = db().prepare('SELECT operation_id, requested_at, checked_at, healthy_at, players FROM restart_rollout WHERE server = ?').get(server) as
{ operation_id: string | null; requested_at: string; healthy_at: string | null; players: number | null } | undefined; { operation_id: string | null; requested_at: string; checked_at: string | null; healthy_at: string | null; players: number | null } | undefined;
db().prepare("UPDATE server_deployments SET state = 'awaiting-health', submitted_at = ? WHERE id = ?").run(submittedAt, id); db().prepare("UPDATE server_deployments SET state = 'awaiting-health', submitted_at = ? WHERE id = ?").run(submittedAt, id);
recordProgress(id, { command: { state: 'attempted', at: submittedAt, message: 'Adopted a previously recorded submission; no command is replayed.' } });
if (slot && ((slot.operation_id && slot.requested_at >= submittedAt) || (slot.requested_at < submittedAt && !slot.healthy_at))) { if (slot && ((slot.operation_id && slot.requested_at >= submittedAt) || (slot.requested_at < submittedAt && !slot.healthy_at))) {
const finishedAt = new Date().toISOString();
const error = 'Existing restart cannot be correlated safely. Inspect the server before retrying.';
db().prepare("UPDATE server_deployments SET state = 'interrupted', finished_at = ?, error = ? WHERE id = ?") db().prepare("UPDATE server_deployments SET state = 'interrupted', finished_at = ?, error = ? WHERE id = ?")
.run(new Date().toISOString(), 'Existing restart cannot be correlated safely. Inspect the server before retrying.', id); .run(finishedAt, error, id);
recordProgress(id, { recovery: { state: 'unknown', message: 'Existing restart recovery could not be correlated with this operation.', operationId: id },
observedAt: null, event: { kind: 'outcome', message: error, details: { state: 'interrupted', finishedAt, error } } });
return; return;
} }
if (slot && slot.requested_at >= submittedAt) { if (slot && slot.requested_at >= submittedAt) {
@@ -96,16 +103,24 @@ function adoptSubmittedRestart(id: string, server: string, submittedAt: string):
} }
if (slot?.healthy_at && slot.requested_at >= submittedAt) { if (slot?.healthy_at && slot.requested_at >= submittedAt) {
db().prepare("UPDATE server_deployments SET state = 'done', finished_at = ?, players = ? WHERE id = ?").run(slot.healthy_at, slot.players, id); db().prepare("UPDATE server_deployments SET state = 'done', finished_at = ?, players = ? WHERE id = ?").run(slot.healthy_at, slot.players, id);
recordProgress(id, { observedAt: slot.healthy_at, event: { kind: 'outcome', message: 'Adopted recorded recovery completion.',
details: { state: 'done', finishedAt: slot.healthy_at, error: null, warnings: [] } } });
} }
deploymentStage(id, 'Adopted existing restart; verifying recovery without restarting again'); deploymentStage(id, 'Adopted existing restart; verifying recovery without restarting again', {
phase: 'verify-container', checks: [], blocker: null, observedAt: slot?.checked_at ?? null,
recovery: { state: slot?.healthy_at && slot.requested_at >= submittedAt ? 'released' : 'held',
at: slot?.healthy_at && slot.requested_at >= submittedAt ? slot.healthy_at : undefined,
message: slot?.healthy_at && slot.requested_at >= submittedAt ? 'Adopted recorded recovery completion.' : 'Observing the previously submitted restart without replaying it.', operationId: id }
});
} }
export function cancelProvisioning(id: string, reason = 'Cancelled by requester'): boolean { export function cancelProvisioning(id: string, reason = 'Cancelled by requester'): boolean {
return db().transaction(() => { return db().transaction(() => {
const finishedAt = new Date().toISOString();
const result = db().prepare(`UPDATE server_deployments SET state = 'cancelled', finished_at = ?, stage = ?, error = NULL const result = db().prepare(`UPDATE server_deployments SET state = 'cancelled', finished_at = ?, stage = ?, error = NULL
WHERE id = ? AND action IN ('start','restart','stop','rebuild') AND state = 'waiting' AND submitted_at IS NULL`) WHERE id = ? AND action IN ('start','restart','stop','rebuild') AND state = 'waiting' AND submitted_at IS NULL`)
.run(new Date().toISOString(), reason, id); .run(finishedAt, reason, id);
if (result.changes) recordDeploymentEvent(id, reason); if (result.changes) recordProgress(id, { blocker: null, checks: [], event: { kind: 'outcome', message: reason, details: { state: 'cancelled', finishedAt, error: null, warnings: [] } } });
return result.changes > 0; return result.changes > 0;
})(); })();
} }
@@ -143,7 +158,8 @@ export function requestLifecycle(server: string, action: RuntimeAction) {
const id = beginDeployment(server, action); const id = beginDeployment(server, action);
db().prepare('UPDATE servers SET desired_state = ? WHERE name = ?').run(action === 'stop' ? 'stopped' : 'running', server); db().prepare('UPDATE servers SET desired_state = ? WHERE name = ?').run(action === 'stop' ? 'stopped' : 'running', server);
db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id); db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id);
deploymentStage(id, action === 'start' ? 'Waiting to start the container' : 'Waiting to verify the server is empty'); deploymentStage(id, action === 'start' ? 'Waiting to start the container' : 'Waiting to verify the server is empty', { phase: 'eligibility', observedAt: null,
blocker: { code: 'eligibility', message: action === 'start' ? 'Waiting for start eligibility and recovery capacity.' : 'Waiting for eligibility and a fresh empty or stopped server.', automatic: true } });
return { ok: true as const, deploymentId: id }; return { ok: true as const, deploymentId: id };
})(); })();
} }
@@ -177,7 +193,8 @@ function queueWatchdogAction(server: string, action: 'restart' | 'rebuild', inci
} else { } else {
id = beginDeployment(server, action, `watchdog:${incidentId}`); id = beginDeployment(server, action, `watchdog:${incidentId}`);
db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id); db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id);
deploymentStage(id, 'Waiting to verify the server is empty'); deploymentStage(id, 'Waiting to verify the server is empty', { phase: 'eligibility', observedAt: null,
blocker: { code: 'eligibility', message: 'Waiting for eligibility and a fresh empty or stopped server.', automatic: true } });
} }
db().prepare("INSERT INTO provisioning_sources (operation_id, kind, source_id, server) VALUES (?, 'watchdog', ?, ?)").run(id, incidentId, server); db().prepare("INSERT INTO provisioning_sources (operation_id, kind, source_id, server) VALUES (?, 'watchdog', ?, ?)").run(id, incidentId, server);
return { ok: true, deploymentId: id }; return { ok: true, deploymentId: id };
@@ -209,8 +226,14 @@ export function requestWatchdogLifecycle(server: string, action: 'restart' | 're
WHERE r.server = ? AND r.healthy_at IS NULL AND d.state IN ('failed','interrupted','cancelled') WHERE r.server = ? AND r.healthy_at IS NULL AND d.state IN ('failed','interrupted','cancelled')
AND a.cycle < i.cycle`).get(incidentId, server) as { operation_id: string } | undefined; AND a.cycle < i.cycle`).get(incidentId, server) as { operation_id: string } | undefined;
if (retained && !restartRequestInFlight(server)) { if (retained && !restartRequestInFlight(server)) {
db().prepare('UPDATE restart_rollout SET operation_id = ? WHERE server = ? AND operation_id = ? AND healthy_at IS NULL') const transferred = db().prepare('UPDATE restart_rollout SET operation_id = ? WHERE server = ? AND operation_id = ? AND healthy_at IS NULL')
.run(next.deploymentId, server, retained.operation_id); .run(next.deploymentId, server, retained.operation_id);
if (transferred.changes) {
recordProgress(retained.operation_id, { recovery: { state: 'transferred', operationId: next.deploymentId,
message: 'Recovery capacity transferred to the explicit new incident attempt.' } });
recordProgress(next.deploymentId, { recovery: { state: 'held', operationId: next.deploymentId,
message: 'Recovery capacity retained from the prior incident attempt.' } });
}
recordDeploymentEvent(retained.operation_id, 'Recovery capacity handed to explicit Watchdog retry ' + next.deploymentId); recordDeploymentEvent(retained.operation_id, 'Recovery capacity handed to explicit Watchdog retry ' + next.deploymentId);
recordDeploymentEvent(next.deploymentId, 'Retained recovery capacity from prior incident attempt ' + retained.operation_id); recordDeploymentEvent(next.deploymentId, 'Retained recovery capacity from prior incident attempt ' + retained.operation_id);
} }
@@ -249,8 +272,16 @@ export function escalateWatchdogRestart(incidentId: string, operationId: string,
const reason = 'Watchdog restart recovery timed out; handed recovery to rebuild ' + replacement.deploymentId; const reason = 'Watchdog restart recovery timed out; handed recovery to rebuild ' + replacement.deploymentId;
if (operation.state === 'awaiting-health') finishDeployment(operationId, { ok: false, message: reason }); if (operation.state === 'awaiting-health') finishDeployment(operationId, { ok: false, message: reason });
else recordDeploymentEvent(operationId, reason); else recordDeploymentEvent(operationId, reason);
if (slot) db().prepare('UPDATE restart_rollout SET operation_id = ?, error = ? WHERE server = ? AND operation_id = ? AND healthy_at IS NULL') if (slot) {
const transferred = db().prepare('UPDATE restart_rollout SET operation_id = ?, error = ? WHERE server = ? AND operation_id = ? AND healthy_at IS NULL')
.run(replacement.deploymentId, reason, operation.server, operationId); .run(replacement.deploymentId, reason, operation.server, operationId);
if (transferred.changes) {
recordProgress(operationId, { recovery: { state: 'transferred', operationId: replacement.deploymentId,
message: 'Recovery capacity transferred to the rebuild operation.' } });
recordProgress(replacement.deploymentId, { recovery: { state: 'held', operationId: replacement.deploymentId,
message: 'Recovery capacity retained from the previous restart.' } });
}
}
recordDeploymentEvent(replacement.deploymentId, 'Recovery handed off from restart ' + operationId + '; waiting for a fresh empty or stopped server'); recordDeploymentEvent(replacement.deploymentId, 'Recovery handed off from restart ' + operationId + '; waiting for a fresh empty or stopped server');
return replacement; return replacement;
})(); })();
+56 -8
View File
@@ -5,6 +5,8 @@ import { probeServerStatus } from '../fleet/status-evidence.js';
import { deploymentStage, finishDeployment } from './history.js'; import { deploymentStage, finishDeployment } from './history.js';
import { applyStagedFilesForOperation } from '../fleet/staged-files.js'; import { applyStagedFilesForOperation } from '../fleet/staged-files.js';
import type { RuntimeAction } from '../../provisioning.js'; import type { RuntimeAction } from '../../provisioning.js';
import type { ProvisioningBlocker } from '../../provisioning-detail.js';
import { recordProgress, type ProgressUpdate } from './progress.js';
// Komodo queues restart work. Don't mistake the still-running old container // Komodo queues restart work. Don't mistake the still-running old container
// for a completed restart when an API kick immediately follows the request. // for a completed restart when an API kick immediately follows the request.
@@ -38,21 +40,28 @@ export function restartSettling(server: string): boolean {
} }
export function rolloutWaitReason(server: string, operationId = '', action: RuntimeAction = 'restart'): string | null { export function rolloutWaitReason(server: string, operationId = '', action: RuntimeAction = 'restart'): string | null {
if (db().prepare(`SELECT 1 FROM server_deployments WHERE server = ? AND id != ? AND server_deleted_at IS NULL return rolloutBlocker(server, operationId, action)?.message ?? null;
}
export function rolloutBlocker(server: string, operationId = '', action: RuntimeAction = 'restart'): ProvisioningBlocker | null {
const conflict = db().prepare(`SELECT id FROM server_deployments WHERE server = ? AND id != ? AND server_deleted_at IS NULL
AND ((action IN ('start','stop','rebuild') AND state IN ('running','awaiting-health')) AND ((action IN ('start','stop','rebuild') AND state IN ('running','awaiting-health'))
OR (action = 'destroy' AND state IN ('waiting','running')))`).get(server, operationId)) { OR (action = 'destroy' AND state IN ('waiting','running')))`).get(server, operationId) as { id: string } | undefined;
return 'Waiting for the current provisioning operation'; if (conflict) {
return { code: 'operation-conflict', message: 'Waiting for the current provisioning operation', automatic: true,
href: `/provisioning/${encodeURIComponent(conflict.id)}`, label: 'Blocking operation' };
} }
const slot = slotFor(server); const slot = slotFor(server);
const ownHandoff = slot?.operation_id === operationId && !!db().prepare(`SELECT 1 FROM server_deployments d const ownHandoff = slot?.operation_id === operationId && !!db().prepare(`SELECT 1 FROM server_deployments d
WHERE d.id = ? AND d.server_deleted_at IS NULL AND d.action IN ('restart','rebuild') AND EXISTS ( WHERE d.id = ? AND d.server_deleted_at IS NULL AND d.action IN ('restart','rebuild') AND EXISTS (
SELECT 1 FROM provisioning_sources s WHERE s.operation_id = d.id AND s.kind = 'watchdog')`).get(operationId); SELECT 1 FROM provisioning_sources s WHERE s.operation_id = d.id AND s.kind = 'watchdog')`).get(operationId);
if (slot?.healthy_at === null && !ownHandoff) { if (slot?.healthy_at === null && !ownHandoff) {
return `Waiting for this server to recover from its previous operation${slot.error ? `: ${slot.error}` : ''}`; return { code: 'previous-recovery', message: `Waiting for this server to recover from its previous operation${slot.error ? `: ${slot.error}` : ''}`, automatic: true,
href: slot.operation_id ? `/provisioning/${encodeURIComponent(slot.operation_id)}` : null, label: 'Previous operation' };
} }
const { n } = db().prepare('SELECT COUNT(*) AS n FROM restart_rollout WHERE healthy_at IS NULL').get() as { n: number }; const { n } = db().prepare('SELECT COUNT(*) AS n FROM restart_rollout WHERE healthy_at IS NULL').get() as { n: number };
return n - (ownHandoff && slot?.healthy_at === null ? 1 : 0) >= fleetSettings().restartConcurrency return n - (ownHandoff && slot?.healthy_at === null ? 1 : 0) >= fleetSettings().restartConcurrency
? `Queued to ${action}; waiting for another server to recover` : null; ? { code: 'restart-capacity', message: `Queued to ${action}; waiting for another server to recover`, automatic: true } : null;
} }
export type RolloutRestartResult = export type RolloutRestartResult =
@@ -71,6 +80,7 @@ export async function restartInRollout(server: string, options: { operationId: s
ON CONFLICT(server) DO UPDATE SET requested_at = excluded.requested_at, ON CONFLICT(server) DO UPDATE SET requested_at = excluded.requested_at,
checked_at = NULL, healthy_at = NULL, players = NULL, error = NULL, operation_id = excluded.operation_id`) checked_at = NULL, healthy_at = NULL, players = NULL, error = NULL, operation_id = excluded.operation_id`)
.run(server, new Date().toISOString(), operationId); .run(server, new Date().toISOString(), operationId);
recordProgress(operationId, { blocker: null, recovery: { state: 'held', message: 'Restart recovery capacity reserved.', operationId } });
return null; return null;
}).immediate(); }).immediate();
if (waitReason) return { kind: 'queued', message: waitReason }; if (waitReason) return { kind: 'queued', message: waitReason };
@@ -81,12 +91,20 @@ export async function restartInRollout(server: string, options: { operationId: s
applyStagedFilesForOperation(operationId, server); applyStagedFilesForOperation(operationId, server);
} catch (err) { } catch (err) {
const message = `File activation failed: ${(err as Error).message}`; const message = `File activation failed: ${(err as Error).message}`;
db().transaction(() => {
db().prepare('DELETE FROM restart_rollout WHERE server = ? AND operation_id = ?').run(server, operationId); db().prepare('DELETE FROM restart_rollout WHERE server = ? AND operation_id = ?').run(server, operationId);
recordProgress(operationId, { recovery: { state: 'released', message: 'Capacity released before any remote command: file activation failed.', operationId } });
finishDeployment(operationId, { ok: false, message }); finishDeployment(operationId, { ok: false, message });
})();
return { kind: 'failed', message }; return { kind: 'failed', message };
} }
db().transaction(() => {
db().prepare('UPDATE server_deployments SET submitted_at = ? WHERE id = ?').run(new Date().toISOString(), operationId); db().prepare('UPDATE server_deployments SET submitted_at = ? WHERE id = ?').run(new Date().toISOString(), operationId);
deploymentStage(operationId, options.action === 'start' ? 'Starting container' : 'Restarting container'); deploymentStage(operationId, options.action === 'start' ? 'Starting container' : 'Restarting container', {
phase: 'container-action', blocker: null, checks: [], checkpoint: { key: 'eligibility', label: 'Eligibility checks passed' },
command: { state: 'attempted', message: `${options.action === 'start' ? 'Start' : 'Restart'} submission attempted.` }
});
})();
requestsInFlight.set(server, operationId); requestsInFlight.set(server, operationId);
db().prepare(`UPDATE servers SET container_state = 'restarting', last_error = NULL, last_reply_snippet = NULL, db().prepare(`UPDATE servers SET container_state = 'restarting', last_error = NULL, last_reply_snippet = NULL,
game_started_at = NULL, game_ended_at = NULL WHERE name = ?`).run(server); game_started_at = NULL, game_ended_at = NULL WHERE name = ?`).run(server);
@@ -94,24 +112,38 @@ export async function restartInRollout(server: string, options: { operationId: s
const result = options.action === 'start' ? await komodoRuntimeAction(server, 'start') : await komodoRestart(server); const result = options.action === 'start' ? await komodoRuntimeAction(server, 'start') : await komodoRestart(server);
if (!db().prepare('SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL').get(operationId)) return { kind: 'failed', message: 'Original server was deleted' }; if (!db().prepare('SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL').get(operationId)) return { kind: 'failed', message: 'Original server was deleted' };
if (result.ok) { if (result.ok) {
db().transaction(() => {
db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ? AND server_deleted_at IS NULL").run(operationId); db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ? AND server_deleted_at IS NULL").run(operationId);
deploymentStage(operationId!, 'Waiting for container and game health (RCON or fresh telemetry)'); deploymentStage(operationId!, 'Waiting for container and game health (RCON or fresh telemetry)', {
phase: 'verify-container', command: { state: 'accepted', message: 'Komodo accepted the request; recovery still requires verification.' },
checkpoint: { key: 'container-action', label: 'Container command accepted' },
blocker: { code: 'recovery-settling', message: 'Waiting for the recovery settling interval before checking health.', automatic: true }
});
})();
return { kind: 'started' }; return { kind: 'started' };
} }
// These failures cannot have queued a remote restart. Other failures // These failures cannot have queued a remote restart. Other failures
// (especially a timeout) may have: retain capacity until recovery. // (especially a timeout) may have: retain capacity until recovery.
db().transaction(() => {
if (result.kind === 'no-name' || result.kind === 'unconfigured') { if (result.kind === 'no-name' || result.kind === 'unconfigured') {
db().prepare('DELETE FROM restart_rollout WHERE server = ? AND operation_id = ?').run(server, operationId); db().prepare('DELETE FROM restart_rollout WHERE server = ? AND operation_id = ?').run(server, operationId);
recordProgress(operationId, { command: { state: 'rejected', message: 'The command could not be submitted: ' + result.message },
recovery: { state: 'released', message: 'Capacity released because no remote command could be submitted.', operationId } });
} else { } else {
db().prepare('UPDATE restart_rollout SET error = ? WHERE server = ? AND operation_id = ?').run(`komodo: ${result.message}`, server, operationId); db().prepare('UPDATE restart_rollout SET error = ? WHERE server = ? AND operation_id = ?').run(`komodo: ${result.message}`, server, operationId);
recordProgress(operationId, { command: { state: 'uncertain', message: 'The command may have reached Komodo: ' + result.message } });
} }
finishDeployment(operationId!, { ok: false, message: `komodo: ${result.message}` }); finishDeployment(operationId!, { ok: false, message: `komodo: ${result.message}` });
})();
return { kind: 'failed', message: `komodo: ${result.message}` }; return { kind: 'failed', message: `komodo: ${result.message}` };
} catch (err) { } catch (err) {
if (!db().prepare('SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL').get(operationId)) return { kind: 'failed', message: 'Original server was deleted' }; if (!db().prepare('SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL').get(operationId)) return { kind: 'failed', message: 'Original server was deleted' };
const message = `komodo: ${(err as Error).message}`; const message = `komodo: ${(err as Error).message}`;
db().transaction(() => {
db().prepare('UPDATE restart_rollout SET error = ? WHERE server = ? AND operation_id = ?').run(message, server, operationId); db().prepare('UPDATE restart_rollout SET error = ? WHERE server = ? AND operation_id = ?').run(message, server, operationId);
recordProgress(operationId, { command: { state: 'uncertain', message: 'The command may have reached Komodo: ' + (err as Error).message } });
finishDeployment(operationId!, { ok: false, message }); finishDeployment(operationId!, { ok: false, message });
})();
return { kind: 'failed', message }; return { kind: 'failed', message };
} finally { } finally {
// Start the grace period at completion, including slow/failed requests. // Start the grace period at completion, including slow/failed requests.
@@ -145,23 +177,38 @@ async function refresh(): Promise<void> {
if (Date.now() - Date.parse(slot.checked_at ?? slot.requested_at) < HEALTH_POLL_MS) continue; if (Date.now() - Date.parse(slot.checked_at ?? slot.requested_at) < HEALTH_POLL_MS) continue;
let error: string | null = null; let error: string | null = null;
let players: number | null = null; let players: number | null = null;
const checks: NonNullable<ProgressUpdate['checks']> = [];
try { try {
const health = await komodoInspectHealthy(slot.server); const health = await komodoInspectHealthy(slot.server);
checks.push({ key: 'container', label: 'Container readiness', result: health.ok ? 'pass' : 'waiting', source: 'komodo', actual: health.state ?? null,
checkedAt: new Date().toISOString(), message: health.ok ? null : health.reason });
if (!health.ok) { if (!health.ok) {
error = `unhealthy: ${health.reason}${health.snippet ? ` snippet=${health.snippet.slice(0, 400)}` : ''}`; error = `unhealthy: ${health.reason}${health.snippet ? ` snippet=${health.snippet.slice(0, 400)}` : ''}`;
} else { } else {
const status = await probeServerStatus(slot.server, slot.requested_at); const status = await probeServerStatus(slot.server, slot.requested_at);
checks.push({ key: 'game', label: 'Game status', result: status.ok ? 'pass' : 'waiting', source: status.ok ? status.source : null,
checkedAt: status.observedAt, receivedAt: status.ok ? status.receivedAt : null, actual: status.ok ? status.players : null,
message: status.ok ? null : status.reason });
if (!status.ok) error = `rcon not answering and no fresh telemetry: ${status.reason}`; if (!status.ok) error = `rcon not answering and no fresh telemetry: ${status.reason}`;
else players = status.players; else players = status.players;
} }
} catch (err) { } catch (err) {
error = `health check: ${(err as Error).message}`; error = `health check: ${(err as Error).message}`;
checks.push({ key: checks.length ? 'game' : 'container', label: checks.length ? 'Game status' : 'Container readiness', result: 'unknown', source: null, message: error });
} }
const now = new Date().toISOString(); const now = new Date().toISOString();
db().transaction(() => {
const checked = db().prepare(`UPDATE restart_rollout SET checked_at = ?, healthy_at = ?, players = ?, error = ? const checked = db().prepare(`UPDATE restart_rollout SET checked_at = ?, healthy_at = ?, players = ?, error = ?
WHERE server = ? AND requested_at = ? AND operation_id IS ? AND healthy_at IS NULL`) WHERE server = ? AND requested_at = ? AND operation_id IS ? AND healthy_at IS NULL`)
.run(now, error ? null : now, players, error, slot.server, slot.requested_at, slot.operation_id); .run(now, error ? null : now, players, error, slot.server, slot.requested_at, slot.operation_id);
if (!checked.changes) continue; if (!checked.changes) return;
if (slot.operation_id) recordProgress(slot.operation_id, {
phase: checks.some(check => check.key === 'game') ? 'verify-game' : 'verify-container', checks, observedAt: now,
blocker: error ? { code: 'recovery-health', message: error, automatic: true } : null,
recovery: { state: error ? 'held' : 'released', at: now, operationId: slot.operation_id,
message: error ? 'Restart capacity remains reserved while recovery is checked.' : 'Container and game recovery verified; restart capacity released.' },
checkpoints: error ? [] : [{ key: 'verify-container', label: 'Container readiness verified', at: now }, { key: 'verify-game', label: 'Game status verified', at: now }]
});
if (slot.operation_id && db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'awaiting-health'").get(slot.operation_id)) { if (slot.operation_id && db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'awaiting-health'").get(slot.operation_id)) {
db().prepare('UPDATE server_deployments SET checked_at = ?, players = ? WHERE id = ?').run(now, players, slot.operation_id); db().prepare('UPDATE server_deployments SET checked_at = ?, players = ? WHERE id = ?').run(now, players, slot.operation_id);
if (error) deploymentStage(slot.operation_id, 'Waiting for health: ' + error); if (error) deploymentStage(slot.operation_id, 'Waiting for health: ' + error);
@@ -170,6 +217,7 @@ async function refresh(): Promise<void> {
finishDeployment(slot.operation_id, { ok: true }); finishDeployment(slot.operation_id, { ok: true });
} }
} }
})();
} }
} }
+80 -29
View File
@@ -5,10 +5,12 @@ import { getServer } from '../fleet/servers.js';
import { deploymentStage, finishDeployment, REMOVING_SERVER } from './history.js'; import { deploymentStage, finishDeployment, REMOVING_SERVER } from './history.js';
import { processCreations, processDestructions, rebuildServer } from './instances.js'; import { processCreations, processDestructions, rebuildServer } from './instances.js';
import { komodoInspectHealthy, komodoRuntimeAction } from '../integrations/komodo.js'; import { komodoInspectHealthy, komodoRuntimeAction } from '../integrations/komodo.js';
import { probePlayerCount } from '../fleet/status-evidence.js'; import { probePlayerEvidence } from '../fleet/status-evidence.js';
import { refreshRestartRollout, restartInRollout, rolloutWaitReason } from './restarts.js'; import { refreshRestartRollout, restartInRollout, rolloutBlocker } from './restarts.js';
import { cancelProvisioning } from './requests.js'; import { cancelProvisioning } from './requests.js';
import type { RuntimeAction } from '../../provisioning.js'; import type { RuntimeAction } from '../../provisioning.js';
import type { ProvisioningBlocker } from '../../provisioning-detail.js';
import { recordProgress } from './progress.js';
const stoppedStates = new Set(['stopped', 'exited', 'created', 'dead']); const stoppedStates = new Set(['stopped', 'exited', 'created', 'dead']);
export const isStopped = (state?: string) => stoppedStates.has(state?.toLowerCase() ?? ''); export const isStopped = (state?: string) => stoppedStates.has(state?.toLowerCase() ?? '');
@@ -18,7 +20,7 @@ const sourcesFor = (id: string) => db().prepare('SELECT kind, source_id FROM pro
const priority = (sources: Source[]) => sources.some(s => s.kind === 'watchdog') ? 2 : sources.some(s => s.kind === 'beta') ? 0 : sources.some(s => s.kind === 'campaign') ? 1 : sources.some(s => s.kind === 'maintenance') ? 3 : 2; const priority = (sources: Source[]) => sources.some(s => s.kind === 'watchdog') ? 2 : sources.some(s => s.kind === 'beta') ? 0 : sources.some(s => s.kind === 'campaign') ? 1 : sources.some(s => s.kind === 'maintenance') ? 3 : 2;
/** Source work may hold or withdraw a request, but never executes it. */ /** Source work may hold or withdraw a request, but never executes it. */
function sourceEligibility(operation: Operation): 'ready' | 'held' | 'cancelled' { function sourceEligibility(operation: Operation, onHold?: (source: Source) => void): 'ready' | 'held' | 'cancelled' {
const sources = sourcesFor(operation.id); const sources = sourcesFor(operation.id);
if (!sources.length) return 'ready'; if (!sources.length) return 'ready';
// Watchdog intent gates every linked source, including its drift campaign. // Watchdog intent gates every linked source, including its drift campaign.
@@ -26,7 +28,7 @@ function sourceEligibility(operation: Operation): 'ready' | 'held' | 'cancelled'
if (!db().prepare("SELECT 1 FROM watchdog_incidents WHERE id = ? AND server = ? AND server_deleted_at IS NULL AND state IN ('open','recovering')").get(source.source_id, operation.server) if (!db().prepare("SELECT 1 FROM watchdog_incidents WHERE id = ? AND server = ? AND server_deleted_at IS NULL AND state IN ('open','recovering')").get(source.source_id, operation.server)
|| getServer(operation.server)?.desired_state === 'stopped') return 'cancelled'; || getServer(operation.server)?.desired_state === 'stopped') return 'cancelled';
const incident = db().prepare('SELECT kind FROM watchdog_incidents WHERE id = ?').get(source.source_id) as { kind: 'health' | 'drift' }; const incident = db().prepare('SELECT kind FROM watchdog_incidents WHERE id = ?').get(source.source_id) as { kind: 'health' | 'drift' };
if (!maintenanceAllowed(incident.kind)) return 'held'; if (!maintenanceAllowed(incident.kind)) { onHold?.(source); return 'held'; }
} }
let held = false; let held = false;
let ready = false; let ready = false;
@@ -36,7 +38,7 @@ function sourceEligibility(operation: Operation): 'ready' | 'held' | 'cancelled'
.get(source.source_id, operation.server) as { restart_state: string; files_changed: number; provisioning_id: string } | undefined; .get(source.source_id, operation.server) as { restart_state: string; files_changed: number; provisioning_id: string } | undefined;
if (row?.provisioning_id !== operation.id || !row.files_changed) continue; if (row?.provisioning_id !== operation.id || !row.files_changed) continue;
if (['pending','failed'].includes(row.restart_state)) ready = true; if (['pending','failed'].includes(row.restart_state)) ready = true;
if (['held','manual'].includes(row.restart_state)) held = true; if (['held','manual'].includes(row.restart_state)) { held = true; onHold?.(source); }
} else if (source.kind === 'beta') { } else if (source.kind === 'beta') {
const row = db().prepare(`SELECT b.state, bs.stage, bs.provisioning_id, p.outcome FROM beta_servers bs JOIN betas b ON b.id = bs.beta_id const row = db().prepare(`SELECT b.state, bs.stage, bs.provisioning_id, p.outcome FROM beta_servers bs JOIN betas b ON b.id = bs.beta_id
LEFT JOIN beta_test_plans p ON p.beta_id = b.id WHERE bs.beta_id = ? AND bs.server = ? AND bs.server_deleted_at IS NULL`) LEFT JOIN beta_test_plans p ON p.beta_id = b.id WHERE bs.beta_id = ? AND bs.server = ? AND bs.server_deleted_at IS NULL`)
@@ -44,12 +46,12 @@ function sourceEligibility(operation: Operation): 'ready' | 'held' | 'cancelled'
if (row?.provisioning_id !== operation.id || !['running','ready-promote'].includes(row.state)) continue; if (row?.provisioning_id !== operation.id || !['running','ready-promote'].includes(row.state)) continue;
if (!['awaiting-empty','restarting','awaiting-health'].includes(row.stage)) continue; if (!['awaiting-empty','restarting','awaiting-health'].includes(row.stage)) continue;
if (!row.outcome || ['active','passed'].includes(row.outcome)) ready = true; if (!row.outcome || ['active','passed'].includes(row.outcome)) ready = true;
if (row.outcome === 'paused') held = true; if (row.outcome === 'paused') { held = true; onHold?.(source); }
} else if (source.kind === 'maintenance') { } else if (source.kind === 'maintenance') {
const row = db().prepare('SELECT state, provisioning_id, kind, origin FROM maintenance_jobs WHERE id = ? AND server_deleted_at IS NULL').get(source.source_id) as { state: string; provisioning_id: string; kind: string; origin: string } | undefined; const row = db().prepare('SELECT state, provisioning_id, kind, origin FROM maintenance_jobs WHERE id = ? AND server_deleted_at IS NULL').get(source.source_id) as { state: string; provisioning_id: string; kind: string; origin: string } | undefined;
if (row?.provisioning_id === operation.id && ['queued','restarting'].includes(row.state)) { if (row?.provisioning_id === operation.id && ['queued','restarting'].includes(row.state)) {
if (getServer(operation.server)?.desired_state === 'stopped') return 'cancelled'; if (getServer(operation.server)?.desired_state === 'stopped') return 'cancelled';
if (!maintenanceAllowed() || (row.kind === 'update' && row.origin === 'automatic' && !maintenanceAllowed('updates'))) return 'held'; if (!maintenanceAllowed() || (row.kind === 'update' && row.origin === 'automatic' && !maintenanceAllowed('updates'))) { onHold?.(source); return 'held'; }
ready = true; ready = true;
} }
} else ready = true; } else ready = true;
@@ -57,26 +59,31 @@ function sourceEligibility(operation: Operation): 'ready' | 'held' | 'cancelled'
return held ? 'held' : ready ? 'ready' : 'cancelled'; return held ? 'held' : ready ? 'ready' : 'cancelled';
} }
function blocker(server: string, id: string, action: RuntimeAction): string | null { function blocker(server: string, id: string, action: RuntimeAction): ProvisioningBlocker | null {
const sources = sourcesFor(id); const sources = sourcesFor(id);
const rank = priority(sources); const rank = priority(sources);
const others = db().prepare(`SELECT id, state, action, rowid FROM server_deployments WHERE server = ? AND id != ? AND server_deleted_at IS NULL const others = db().prepare(`SELECT id, state, action, rowid FROM server_deployments WHERE server = ? AND id != ? AND server_deleted_at IS NULL
AND state IN ('queued','waiting','running','awaiting-health')`).all(server, id) as { id: string; state: string; action: string; rowid: number }[]; AND state IN ('queued','waiting','running','awaiting-health')`).all(server, id) as { id: string; state: string; action: string; rowid: number }[];
const own = db().prepare('SELECT rowid FROM server_deployments WHERE id = ?').get(id) as { rowid: number }; const own = db().prepare('SELECT rowid FROM server_deployments WHERE id = ?').get(id) as { rowid: number };
if (others.some(other => other.state !== 'waiting' || other.action === 'destroy' const conflict = others.find(other => other.state !== 'waiting' || other.action === 'destroy'
|| priority(sourcesFor(other.id)) < rank || (priority(sourcesFor(other.id)) === rank && other.rowid < own.rowid))) { || priority(sourcesFor(other.id)) < rank || (priority(sourcesFor(other.id)) === rank && other.rowid < own.rowid));
return 'Waiting for another provisioning operation on this server'; if (conflict) {
return { code: 'operation-conflict', message: 'Waiting for another provisioning operation on this server', automatic: true,
href: `/provisioning/${encodeURIComponent(conflict.id)}`, label: 'Blocking operation' };
} }
// Beta activation must not wait on the campaign rows that the beta itself // Beta activation must not wait on the campaign rows that the beta itself
// holds. Campaign/maintenance callers likewise never block on their own job. // holds. Campaign/maintenance callers likewise never block on their own job.
if (rank > 0 && db().prepare(`SELECT 1 FROM beta_servers bs JOIN betas b ON b.id = bs.beta_id const beta = rank > 0 ? db().prepare(`SELECT b.id FROM beta_servers bs JOIN betas b ON b.id = bs.beta_id
WHERE bs.server = ? AND bs.server_deleted_at IS NULL AND b.state IN ('running','ready-promote') LIMIT 1`).get(server)) return 'Waiting for beta work on this server'; WHERE bs.server = ? AND bs.server_deleted_at IS NULL AND b.state IN ('running','ready-promote') LIMIT 1`).get(server) as { id: string } | undefined : undefined;
if (rank > 1 && db().prepare(`SELECT 1 FROM deployment_servers ds WHERE ds.server = ? AND ds.server_deleted_at IS NULL if (beta) return { code: 'beta-work', message: 'Waiting for beta work on this server', automatic: true, href: `/betas/${encodeURIComponent(beta.id)}`, label: 'Blocking beta' };
const campaign = rank > 1 ? db().prepare(`SELECT ds.deployment_id FROM deployment_servers ds WHERE ds.server = ? AND ds.server_deleted_at IS NULL
AND ds.restart_state IN ('pending','failed','held') AND (ds.provisioning_id IS NULL OR EXISTS ( AND ds.restart_state IN ('pending','failed','held') AND (ds.provisioning_id IS NULL OR EXISTS (
SELECT 1 FROM server_deployments d WHERE d.id = ds.provisioning_id AND d.state IN ('queued','waiting','running','awaiting-health'))) SELECT 1 FROM server_deployments d WHERE d.id = ds.provisioning_id AND d.state IN ('queued','waiting','running','awaiting-health')))
AND NOT EXISTS (SELECT 1 FROM watchdog_actions a WHERE a.operation_id = ? AND a.campaign_id = ds.deployment_id) AND NOT EXISTS (SELECT 1 FROM watchdog_actions a WHERE a.operation_id = ? AND a.campaign_id = ds.deployment_id)
LIMIT 1`).get(server, id)) return 'Waiting for campaign work on this server'; LIMIT 1`).get(server, id) as { deployment_id: string } | undefined : undefined;
return rolloutWaitReason(server, id, action); if (campaign) return { code: 'campaign-work', message: 'Waiting for campaign work on this server', automatic: true,
href: `/campaigns/${encodeURIComponent(campaign.deployment_id)}`, label: 'Blocking campaign' };
return rolloutBlocker(server, id, action);
} }
interface Operation { id: string; server: string; action: RuntimeAction; state: string; stage: string } interface Operation { id: string; server: string; action: RuntimeAction; state: string; stage: string }
@@ -100,7 +107,9 @@ async function run(): Promise<void> {
try { await processOperation(operation); } try { await processOperation(operation); }
catch (err) { catch (err) {
// Observation failures are retried; remote mutations are never replayed. // Observation failures are retried; remote mutations are never replayed.
deploymentStage(operation.id, 'Waiting for verification: ' + (err as Error).message); deploymentStage(operation.id, 'Waiting for verification: ' + (err as Error).message, {
checks: [], blocker: { code: 'observation-unavailable', message: (err as Error).message, automatic: true }
});
} }
} }
} }
@@ -112,36 +121,65 @@ async function processOperation(operation: Operation): Promise<void> {
// Watchdog is paused; never strand the reserved replacement identity. // Watchdog is paused; never strand the reserved replacement identity.
if (action === 'rebuild' && operation.stage === REMOVING_SERVER) { await rebuildServer(id); return; } if (action === 'rebuild' && operation.stage === REMOVING_SERVER) { await rebuildServer(id); return; }
if (operation.state === 'waiting') { if (operation.state === 'waiting') {
const eligibility = sourceEligibility(operation); const heldBy: Source[] = [];
const eligibility = sourceEligibility(operation, source => heldBy.push(source));
if (eligibility === 'cancelled') { cancelProvisioning(id, 'Requester cancelled or replaced this operation'); return; } if (eligibility === 'cancelled') { cancelProvisioning(id, 'Requester cancelled or replaced this operation'); return; }
if (eligibility === 'held') { deploymentStage(id, 'Waiting for the requester to resume'); return; } if (eligibility === 'held') {
const source = heldBy[0];
const href = source ? source.kind === 'campaign' ? `/campaigns/${encodeURIComponent(source.source_id)}`
: source.kind === 'beta' ? `/betas/${encodeURIComponent(source.source_id)}`
: source.kind === 'watchdog' ? `/maintenance/${encodeURIComponent(source.source_id)}`
: `/maintenance?view=activity#job-${encodeURIComponent(source.source_id)}` : null;
deploymentStage(id, 'Waiting for the requester to resume', { phase: 'eligibility', checks: [],
blocker: { code: 'requester-held', message: 'Waiting for the requester to resume', automatic: false, href, label: 'Requesting work' } });
return;
}
} }
if (!getServer(server)) { finishDeployment(id, { ok: false, message: 'Server is no longer registered.' }); return; } if (!getServer(server)) { finishDeployment(id, { ok: false, message: 'Server is no longer registered.' }); return; }
const watchdog = sourcesFor(id).some(source => source.kind === 'watchdog'); const watchdog = sourcesFor(id).some(source => source.kind === 'watchdog');
if (!komodoConfigured() || (!rconConfigured() && !watchdog)) { deploymentStage(id, 'Waiting for Komodo and RCON configuration'); return; } if (!komodoConfigured() || (!rconConfigured() && !watchdog)) { deploymentStage(id, 'Waiting for Komodo and RCON configuration', {
phase: 'eligibility', checks: [], blocker: { code: 'configuration', message: 'Waiting for Komodo and RCON configuration', automatic: false, href: '/settings', label: 'Integration settings' }
}); return; }
if (operation.state === 'awaiting-health') { if (operation.state === 'awaiting-health') {
const health = await komodoInspectHealthy(server); const health = await komodoInspectHealthy(server);
if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'awaiting-health'").get(id)) return;
recordProgress(id, { phase: 'verify-stopped', checks: [{ key: 'stopped', label: 'Container stopped', source: 'komodo',
result: isStopped(health.state) ? 'pass' : health.state ? 'waiting' : 'unknown', actual: health.state ?? null,
message: health.ok ? null : health.reason }], blocker: isStopped(health.state) ? null
: { code: 'stopped-state', message: 'Waiting for an explicit stopped container state.', automatic: true } });
if (isStopped(health.state)) { if (isStopped(health.state)) {
db().transaction(() => {
db().prepare("UPDATE servers SET container_state = ?, last_players = NULL, last_roster = NULL, game_started_at = NULL, game_ended_at = NULL WHERE name = ?").run(health.state, server); db().prepare("UPDATE servers SET container_state = ?, last_players = NULL, last_roster = NULL, game_started_at = NULL, game_ended_at = NULL WHERE name = ?").run(health.state, server);
deploymentStage(id, 'Verified container stopped'); deploymentStage(id, 'Verified container stopped', { checkpoint: { key: 'verify-stopped', label: 'Stopped container state verified' } });
finishDeployment(id, { ok: true }); finishDeployment(id, { ok: true });
})();
} else deploymentStage(id, 'Waiting for container to stop' + (!health.ok ? ': ' + health.reason : '')); } else deploymentStage(id, 'Waiting for container to stop' + (!health.ok ? ': ' + health.reason : ''));
return; return;
} }
const reason = blocker(server, id, action); const reason = blocker(server, id, action);
if (reason) { deploymentStage(id, reason); return; } if (reason) { deploymentStage(id, reason.message, { phase: 'eligibility', checks: [], blocker: reason }); return; }
const health = await komodoInspectHealthy(server); const health = await komodoInspectHealthy(server);
if (action !== 'start' && !isStopped(health.state)) { if (action !== 'start' && !isStopped(health.state)) {
const players = await probePlayerCount(server); const evidence = await probePlayerEvidence(server);
if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'waiting'").get(id)) return;
const players = evidence.players;
recordProgress(id, { phase: 'eligibility', checks: [{ key: 'occupancy', label: 'Player occupancy', result: players === 0 ? 'pass' : players === null ? 'unknown' : 'waiting',
source: evidence.source, checkedAt: evidence.observedAt, receivedAt: evidence.receivedAt, expected: 0, actual: players }],
blocker: players === 0 ? null : { code: players === null ? 'players-unknown' : 'players-present',
message: players === null ? 'Waiting for a fresh player count' : `Waiting for ${players} player(s) to leave`, automatic: true } });
db().prepare('UPDATE server_deployments SET players = ?, checked_at = ? WHERE id = ?').run(players, new Date().toISOString(), id); db().prepare('UPDATE server_deployments SET players = ?, checked_at = ? WHERE id = ?').run(players, new Date().toISOString(), id);
if (players !== 0) { if (players !== 0) {
deploymentStage(id, players === null ? 'Waiting for a fresh player count' : `Waiting for ${players} player(s) to leave`); deploymentStage(id, players === null ? 'Waiting for a fresh player count' : `Waiting for ${players} player(s) to leave`);
return; return;
} }
} else if (action !== 'start') {
if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'waiting'").get(id)) return;
recordProgress(id, { phase: 'eligibility', blocker: null, checks: [{ key: 'stopped', label: 'Stopped container permits disruption',
source: 'komodo', result: 'pass', actual: health.state ?? null }] });
} }
// Requests can arrive during the probes. Recheck before claiming the action. // Requests can arrive during the probes. Recheck before claiming the action.
const latestReason = blocker(server, id, action); const latestReason = blocker(server, id, action);
if (latestReason) { deploymentStage(id, latestReason); return; } if (latestReason) { deploymentStage(id, latestReason.message, { phase: 'eligibility', blocker: latestReason }); return; }
if (sourceEligibility(operation) !== 'ready') return; if (sourceEligibility(operation) !== 'ready') return;
if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'waiting'").get(id)) return; if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'waiting'").get(id)) return;
if (action === 'rebuild') { await rebuildServer(id); return; } if (action === 'rebuild') { await rebuildServer(id); return; }
@@ -155,13 +193,26 @@ async function processOperation(operation: Operation): Promise<void> {
} }
return; return;
} }
db().transaction(() => {
db().prepare('UPDATE server_deployments SET submitted_at = ? WHERE id = ?').run(new Date().toISOString(), id); db().prepare('UPDATE server_deployments SET submitted_at = ? WHERE id = ?').run(new Date().toISOString(), id);
deploymentStage(id, 'Stopping container'); deploymentStage(id, 'Stopping container', { phase: 'container-action', blocker: null, checks: [],
checkpoint: { key: 'eligibility', label: 'Eligibility checks passed' }, command: { state: 'attempted', message: 'Stop submission attempted.' } });
})();
// Persist intent first; a lost reply must not issue another stop later. // Persist intent first; a lost reply must not issue another stop later.
try { try {
const result = await komodoRuntimeAction(server, 'stop'); const result = await komodoRuntimeAction(server, 'stop');
if (!result.ok) { finishDeployment(id, { ok: false, message: result.message }); return; } if (!result.ok) {
db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ?").run(id); recordProgress(id, { command: { state: result.kind === 'no-name' || result.kind === 'unconfigured' ? 'rejected' : 'uncertain', message: result.message } });
deploymentStage(id, 'Waiting for container to stop'); finishDeployment(id, { ok: false, message: result.message }); return;
} catch (err) { finishDeployment(id, { ok: false, message: (err as Error).message }); } }
db().transaction(() => {
db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ?").run(id);
deploymentStage(id, 'Waiting for container to stop', { phase: 'verify-stopped',
checkpoint: { key: 'container-action', label: 'Stop command accepted' },
command: { state: 'accepted', message: 'Komodo accepted the stop request; stopped state still requires verification.' } });
})();
} catch (err) {
recordProgress(id, { command: { state: 'uncertain', message: (err as Error).message } });
finishDeployment(id, { ok: false, message: (err as Error).message });
}
} }
+3 -3
View File
@@ -1,11 +1,11 @@
<script lang="ts"> <script lang="ts">
import { timeFormat } from '$ui/stores/time-zone.js'; import { timeFormat } from '$ui/stores/time-zone.js';
import type { LaunchRecipeSummary } from '$lib/deployment-plan.js'; import type { LaunchRecipeSummary } from '$lib/deployment-plan.js';
let { recipe, historical = false }: { recipe: LaunchRecipeSummary; historical?: boolean } = $props(); let { recipe, historical = false, selectionStatus = null }: { recipe: LaunchRecipeSummary; historical?: boolean; selectionStatus?: 'queued' | 'attempted' | null } = $props();
</script> </script>
<div class="recipe"> <div class="recipe">
<div class="recipe-heading"><div><span class="eyebrow">{historical ? 'Recorded launch files' : 'Current promoted setup'}</span><h3>{recipe.mapName}</h3></div><span class="game">{recipe.game === 't6' ? 'Black Ops II' : 'Black Ops I'}</span></div> <div class="recipe-heading"><div><span class="eyebrow">{historical ? selectionStatus === 'queued' ? 'Queued setup capture' : selectionStatus === 'attempted' ? 'Selected for this attempt' : 'Recorded launch files' : 'Current promoted setup'}</span><h3>{recipe.mapName}</h3></div><span class="game">{recipe.game === 't6' ? 'Black Ops II' : 'Black Ops I'}</span></div>
<dl> <dl>
<div><dt>Configuration</dt><dd>{recipe.configSource}</dd></div> <div><dt>Configuration</dt><dd>{recipe.configSource}</dd></div>
<div><dt>Container image</dt><dd class="mono">{recipe.image}</dd></div> <div><dt>Container image</dt><dd class="mono">{recipe.image}</dd></div>
@@ -14,7 +14,7 @@
<details> <details>
<summary>{recipe.files.length} managed file{recipe.files.length === 1 ? '' : 's'} {historical ? 'recorded for this deployment' : 'in the current setup'}</summary> <summary>{recipe.files.length} managed file{recipe.files.length === 1 ? '' : 's'} {historical ? 'recorded for this deployment' : 'in the current setup'}</summary>
<ul>{#each recipe.files as file}<li><div><strong>{file.name}</strong><span>{file.version !== null ? `v${file.version}` : 'Generated'} · {file.kind} · {file.size.toLocaleString()} bytes</span></div><code>{file.path}</code><small title={file.sha256}>SHA-256 {file.sha256.slice(0, 12)}</small></li>{/each}</ul> <ul>{#each recipe.files as file}<li><div><strong>{file.name}</strong><span>{file.version !== null ? `v${file.version}` : 'Generated'} · {file.kind} · {file.size.toLocaleString()} bytes</span></div><code>{file.path}</code><small title={file.sha256}>SHA-256 {file.sha256.slice(0, 12)}</small></li>{/each}</ul>
<p>{historical ? 'Recorded' : 'Previewed'} {$timeFormat.dateTime(recipe.capturedAt)}. {historical ? 'This records the files selected for this deployment. Launch again uses the latest promoted setup for this map.' : 'This preview shows the promoted setup at the time above. The deployment history records the files selected when provisioning begins.'}</p> <p>{historical ? 'Recorded' : 'Previewed'} {$timeFormat.dateTime(recipe.capturedAt)}. {historical ? selectionStatus === 'queued' ? 'Versions are selected again when creation begins. This queued capture does not pin them.' : selectionStatus === 'attempted' ? 'These files were selected for this attempt; this does not confirm installation succeeded. Launch again uses the latest promoted setup.' : 'This records the saved setup for this deployment. Launch again uses the latest promoted setup for this map.' : 'This preview shows the promoted setup at the time above. The deployment history records the files selected when provisioning begins.'}</p>
<small class="fingerprint" title={recipe.fingerprint}>Recipe {recipe.fingerprint.slice(0, 16)}</small> <small class="fingerprint" title={recipe.fingerprint}>Recipe {recipe.fingerprint.slice(0, 16)}</small>
</details> </details>
</div> </div>
@@ -0,0 +1,132 @@
<script lang="ts">
import { onDestroy } from 'svelte';
import type { ProvisioningDetail, ProvisioningEvent } from '$lib/provisioning-detail.js';
import { timeFormat } from '$ui/stores/time-zone.js';
import { isProvisioningEventPage } from '$ui/provisioning-presentation.js';
import Button from './primitives/Button.svelte';
let { detail }: { detail: ProvisioningDetail } = $props();
let open = $state(false);
let query = $state('');
let order = $state<'newest' | 'oldest'>('newest');
let appliedQuery = $state('');
let appliedOrder = $state<'newest' | 'oldest'>('newest');
let entries = $state<ProvisioningEvent[]>([]);
let frozenRecent = $state<ProvisioningEvent[] | null>(null);
let total = $state(0);
let baselineTotal = $state<number | null>(null);
let watermark = $state<number | null>(null);
let cursor = $state<number | null>(null);
let loaded = $state(false);
let loading = $state(false);
let error = $state('');
let requestNumber = 0;
let controller: AbortController | null = null;
let disposed = false;
const recent = $derived((frozenRecent ?? detail.events).slice().sort((a, b) => b.id - a.id).slice(0, 5));
const newEvents = $derived(baselineTotal === null ? 0 : Math.max(0, detail.activity.total - baselineTotal));
async function load(more = false) {
if (more && (loading || cursor === null)) return;
controller?.abort();
controller = new AbortController();
const currentController = controller;
const request = ++requestNumber;
const id = detail.deployment.id;
const current = () => !disposed && request === requestNumber && detail.deployment.id === id;
loading = true; error = '';
const requestedQuery = more ? appliedQuery : query;
const requestedOrder = more ? appliedOrder : order;
const params = new URLSearchParams({ order: requestedOrder, limit: '25' });
if (requestedQuery) params.set('query', requestedQuery);
if (more && cursor !== null && watermark !== null) {
params.set('cursor', String(cursor));
params.set('watermark', String(watermark));
}
let timedOut = false;
const timeout = setTimeout(() => { timedOut = true; currentController.abort(); }, 15_000);
try {
const response = await fetch(`/api/provisioning/${encodeURIComponent(id)}/events?${params}`, { signal: currentController.signal, cache: 'no-store' });
if (!response.ok) throw new Error('Could not load activity history.');
const body: unknown = await response.json();
if (!isProvisioningEventPage(body)) throw new Error('The activity response was incomplete.');
if (!current()) return;
entries = more ? [...entries, ...body.events.filter(event => !entries.some(previous => previous.id === event.id))] : body.events;
total = body.total; watermark = body.watermark; cursor = body.nextCursor; loaded = true;
if (!more) { baselineTotal = body.totalEvents; appliedQuery = requestedQuery; appliedOrder = requestedOrder; }
} catch (reason) {
if (current() && (!currentController.signal.aborted || timedOut)) error = timedOut ? 'Activity refresh timed out. Try again.' : reason instanceof Error ? reason.message : 'Could not load activity history.';
} finally {
clearTimeout(timeout);
if (current()) loading = false;
}
}
function toggle(event: Event) {
const expanded = (event.currentTarget as HTMLDetailsElement).open;
open = expanded;
if (expanded) {
frozenRecent = detail.events.slice();
if (!loaded && !loading) void load();
} else frozenRecent = null;
}
function jumpToLatest() {
query = appliedQuery; order = 'newest';
frozenRecent = detail.events.slice();
void load();
}
onDestroy(() => { disposed = true; requestNumber++; controller?.abort(); });
</script>
<section aria-labelledby="activity-title" class="activity">
<div class="heading"><div><span class="eyebrow">Activity</span><h2 id="activity-title">Recent updates</h2></div><span class="count">{detail.activity.total} event{detail.activity.total === 1 ? '' : 's'}</span></div>
<p class="muted">{open ? 'Recent updates stay in place while you read the full history.' : 'Newest first · updates automatically'}</p>
{#if recent.length}<ol class="timeline recent">
{#each recent as event (event.id)}<li><time datetime={event.created_at}>{$timeFormat.dateTime(event.created_at)}</time><p>{event.message}</p></li>{/each}
</ol>{:else}<p class="empty">No detailed activity was recorded. The recorded outcome and timestamps are preserved above.</p>{/if}
<details class="history" ontoggle={toggle}>
<summary>Full activity history <span>{detail.activity.total} events</span></summary>
<div class="history-content">
{#if newEvents > 0}<div class="new-events" role="status"><span>{newEvents} new event{newEvents === 1 ? '' : 's'} since this view was loaded.</span><Button size="sm" onclick={jumpToLatest}>Jump to latest</Button></div>{/if}
<form class="tools" onsubmit={event => { event.preventDefault(); void load(); }}>
<label class="search">Search activity<input type="search" bind:value={query} placeholder="Find a stage or message" /></label>
<label>Order<select bind:value={order} onchange={() => { void load(); }}><option value="newest">Newest first</option><option value="oldest">Oldest first</option></select></label>
<Button type="submit" disabled={loading}>Search</Button>
</form>
{#if error}<div class="history-error" role="alert"><p>{error} {loaded ? 'Your previous history view is still shown.' : ''}</p><Button size="sm" onclick={() => load()}>Retry history</Button></div>{/if}
{#if loaded}<p class="muted">{appliedOrder === 'newest' ? 'Newest first' : 'Oldest first'}{appliedQuery ? ` · Matching “${appliedQuery}”` : ' · All messages'}</p>{/if}
{#if loading && !loaded}<p class="muted" role="status">Loading activity…</p>
{:else if loaded && !entries.length}<p class="empty">{appliedQuery ? 'No activity matches this search.' : 'No detailed activity recorded.'}</p>
{:else}<ol class="timeline">
{#each entries as event (event.id)}<li><time datetime={event.created_at}>{$timeFormat.dateTime(event.created_at)}</time><p>{event.message}</p></li>{/each}
</ol>{/if}
{#if loaded}<div class="footer"><span>Showing {entries.length} of {total} matching events</span>{#if cursor !== null}<Button size="sm" loading={loading} disabled={loading} onclick={() => load(true)}>Show more activity</Button>{/if}</div>{/if}
</div>
</details>
</section>
<style>
.activity { display: grid; gap: var(--space-4); min-width: 0; }
.heading { display: flex; justify-content: space-between; gap: var(--space-4); align-items: end; flex-wrap: wrap; }
.eyebrow { display: block; font-size: var(--text-xs); color: var(--text-tertiary); margin-bottom: var(--space-2); }
h2 { margin: 0; font-size: var(--text-lg); letter-spacing: var(--tracking-tight); }
p { margin: 0; line-height: 1.65; overflow-wrap: anywhere; font-size: var(--text-sm); }
.muted, .count, time, .footer { font-size: var(--text-xs); color: var(--text-secondary); }
.empty { padding: var(--space-4); border: 1px solid var(--border-subtle); border-radius: var(--radius-md); color: var(--text-secondary); }
.timeline { list-style: none; margin: 0 0 0 var(--space-1); padding: 0 0 0 var(--space-5); border-left: 1px solid var(--border-strong); }
.timeline li { padding: 0 0 var(--space-5); position: relative; }.timeline li:last-child { padding-bottom: 0; }
.timeline li::before { content: ''; position: absolute; top: 6px; left: -24px; width: 7px; height: 7px; background: var(--border-control); border-radius: var(--radius-pill); }
.recent li:first-child::before { background: var(--accent-primary); }.timeline p { margin-top: var(--space-1); }
.history { border: 1px solid var(--border-subtle); border-radius: var(--radius-md); background: var(--surface); }
summary { cursor: pointer; padding: var(--space-5); font-size: var(--text-sm); font-weight: var(--weight-semibold); overflow-wrap: anywhere; }
summary span { margin-left: var(--space-2); color: var(--text-tertiary); font-size: var(--text-xs); font-weight: var(--weight-regular); }
summary:focus-visible { outline: 2px solid var(--border-focus); outline-offset: -3px; border-radius: var(--radius-sm); }
.history-content { padding: 0 var(--space-5) var(--space-5); display: grid; gap: var(--space-5); }
.tools { display: flex; flex-wrap: wrap; align-items: end; gap: var(--space-3); }.tools label { display: grid; gap: var(--space-2); font-size: var(--text-xs); color: var(--text-secondary); }.search { flex: 1 1 160px; }
input, select { background: var(--surface-sunken); color: var(--text-primary); border: 1px solid var(--border-control); border-radius: var(--radius-sm); padding: 10px; font: inherit; min-height: 40px; width: 100%; }
input:focus-visible, select:focus-visible { outline: 2px solid var(--border-focus); outline-offset: 2px; }
.new-events { display: flex; align-items: center; gap: var(--space-3); flex-wrap: wrap; border: 1px solid var(--border-strong); border-radius: var(--radius-sm); padding: var(--space-3); color: var(--text-secondary); font-size: var(--text-sm); }.new-events span { flex: 1; }
.footer { display: flex; justify-content: space-between; align-items: center; gap: var(--space-3); flex-wrap: wrap; line-height: 1.6; }
.history-error { display: grid; justify-items: start; gap: var(--space-3); color: var(--state-danger); }
@media (max-width: 600px) { summary { padding: var(--space-4); }.history-content { padding: 0 var(--space-4) var(--space-4); }.tools { align-items: stretch; }.tools .search { flex-basis: 100%; } }
</style>
@@ -0,0 +1,123 @@
<script lang="ts">
import type { ProvisioningDetail } from '$lib/provisioning-detail.js';
import { provisioningGroups, progressIcons, progressStateLabels } from '$ui/provisioning-presentation.js';
import { timeFormat } from '$ui/stores/time-zone.js';
import Badge from './primitives/Badge.svelte';
let { detail }: { detail: ProvisioningDetail } = $props();
const groups = $derived(provisioningGroups(detail));
const progress = $derived(detail.progress);
let expanded = $state<Record<string, boolean>>({});
const evidenceLabels = { pass: 'Passed', waiting: 'Waiting', fail: 'Failed', unknown: 'Unknown' };
const sources = { komodo: 'Komodo', rcon: 'RCON', telemetry: 'Telemetry', combined: 'RCON and telemetry' };
</script>
<section class="execution" aria-labelledby="execution-title">
<div class="section-heading"><div><span class="eyebrow">Execution</span><h2 id="execution-title">Progress and evidence</h2></div></div>
{#if !progress}
<div class="legacy"><h3>{detail.deployment.stage}</h3><p>Detailed checkpoints were not recorded for this operation. Its recorded outcome and original activity remain available below.</p></div>
{:else}
<ol class="phases">
{#each groups as group (group.key)}
<li class:current={group.state === 'current' || group.state === 'waiting'} class:failed={group.state === 'failed'}>
<details open={expanded[group.key] ?? ['current', 'waiting', 'failed'].includes(group.state)}>
<summary
onclick={event => { event.preventDefault(); expanded[group.key] = !(expanded[group.key] ?? ['current', 'waiting', 'failed'].includes(group.state)); }}
onkeydown={event => {
if (event.key !== 'Enter' && event.key !== ' ') return;
event.preventDefault();
if (!event.repeat) expanded[group.key] = !(expanded[group.key] ?? ['current', 'waiting', 'failed'].includes(group.state));
}}
>
<span class="phase-icon state-{group.state}" aria-hidden="true">{progressIcons[group.state]}</span>
<span class="phase-title">{group.label}</span>
<span class="phase-state state-{group.state}">{progressStateLabels[group.state]}</span>
</summary>
<ol class="steps">
{#each group.steps as step (step.key)}
<li aria-current={step.state === 'current' || step.state === 'waiting' ? 'step' : undefined}>
<span class="step-icon state-{step.state}" aria-hidden="true">{progressIcons[step.state]}</span>
<div><strong>{step.label}</strong><span class="step-meta state-{step.state}">{progressStateLabels[step.state]}{#if step.at} · <time datetime={step.at}>{$timeFormat.dateTime(step.at)}</time>{/if}</span></div>
</li>
{/each}
</ol>
</details>
</li>
{/each}
</ol>
{#if progress.command}
<div class="command" class:uncertain={progress.command.state === 'uncertain'}>
<h3>{progress.command.state === 'attempted' ? 'Command submission attempted' : progress.command.state === 'accepted' ? 'Remote request accepted' : progress.command.state === 'rejected' ? 'Remote request rejected' : 'Remote response uncertain'}</h3>
<p>{progress.command.message}</p>
<time datetime={progress.command.at}>{$timeFormat.dateTime(progress.command.at)}</time>
</div>
{/if}
{#if progress.checkpoints.some(checkpoint => !groups.some(group => group.steps.some(step => step.key === checkpoint.key)))}
<details class="milestones"><summary>Recorded checkpoints</summary><ul>
{#each progress.checkpoints.filter(checkpoint => !groups.some(group => group.steps.some(step => step.key === checkpoint.key))) as checkpoint (checkpoint.key)}
<li><strong>{checkpoint.label}</strong><time datetime={checkpoint.at}>{$timeFormat.dateTime(checkpoint.at)}</time></li>
{/each}
</ul></details>
{/if}
<div class="evidence-heading"><h3>Latest recorded checks</h3>{#if progress.observedAt}<span>Evaluated <time datetime={progress.observedAt}>{$timeFormat.dateTime(progress.observedAt)}</time></span>{/if}</div>
{#if progress.checks.length}
<div class="checks">
{#each progress.checks as check (check.key)}
<article class="check">
<div class="check-heading"><strong>{check.label}</strong><Badge variant={check.result === 'pass' ? 'online' : check.result === 'fail' ? 'danger' : check.result === 'waiting' ? 'warning' : 'neutral'}>{evidenceLabels[check.result]}</Badge></div>
{#if check.expected !== null || check.actual !== null}<dl>
{#if check.expected !== null}<div><dt>Expected</dt><dd>{check.expected}</dd></div>{/if}
<div><dt>Observed</dt><dd>{check.actual ?? 'Unknown'}</dd></div>
</dl>{/if}
{#if check.message}<p>{check.message}</p>{/if}
<div class="evidence-meta"><span>{check.source ? sources[check.source] : 'Source not recorded'}</span><span>Checked <time datetime={check.checkedAt}>{$timeFormat.dateTime(check.checkedAt)}</time></span>{#if check.receivedAt}<span>Evidence received <time datetime={check.receivedAt}>{$timeFormat.dateTime(check.receivedAt)}</time></span>{/if}</div>
</article>
{/each}
</div>
{:else}<p class="empty">No check evidence has been recorded for this operation yet.</p>{/if}
{#if detail.deployment.expected_map}<p class="map-note">Selected engine map: <code>{detail.deployment.expected_map}</code>. Engine-map verification does not independently verify a variant or start location.</p>{/if}
{/if}
</section>
<style>
.execution { display: grid; gap: var(--space-5); min-width: 0; }
h2, h3, p { margin: 0; }
h2 { font-size: var(--text-lg); letter-spacing: var(--tracking-tight); }
h3 { font-size: var(--text-sm); font-weight: var(--weight-semibold); }
.eyebrow { display: block; margin-bottom: var(--space-2); font-size: var(--text-xs); color: var(--text-tertiary); }
.phases, .steps, .milestones ul { list-style: none; padding: 0; margin: 0; }
.phases { display: grid; gap: var(--space-3); }
.phases > li { border: 1px solid var(--border-subtle); border-radius: var(--radius-md); background: var(--surface); overflow: hidden; }
.phases > li.current { border-color: color-mix(in srgb, var(--accent-primary) 60%, var(--border-subtle)); }
.phases > li.failed { border-color: color-mix(in srgb, var(--state-danger) 60%, var(--border-subtle)); }
summary { cursor: pointer; }
.phases summary { display: flex; align-items: center; gap: var(--space-3); padding: var(--space-5); list-style: none; }
.phases summary::-webkit-details-marker { display: none; }
.phases summary::after { content: '+'; color: var(--text-tertiary); font-size: 18px; }
.phases details[open] > summary::after { content: '−'; }
summary:focus-visible { outline: 2px solid var(--border-focus); outline-offset: -3px; border-radius: var(--radius-sm); }
.phase-icon { flex: 0 0 26px; display: grid; place-items: center; width: 26px; height: 26px; border: 1px solid var(--border-strong); border-radius: var(--radius-pill); font-size: var(--text-sm); }
.phase-title { flex: 1; font-size: var(--text-sm); font-weight: var(--weight-semibold); }
.phase-state { font-size: var(--text-xs); }
.steps { border-top: 1px solid var(--border-subtle); padding: var(--space-4) var(--space-5); display: grid; gap: var(--space-4); }
.steps li { display: flex; align-items: flex-start; gap: var(--space-3); }
.step-icon { flex: 0 0 26px; text-align: center; line-height: 1.5; }
.steps strong { display: block; font-size: var(--text-sm); font-weight: var(--weight-medium); line-height: 1.5; }
.step-meta { display: block; font-size: var(--text-xs); margin-top: var(--space-1); line-height: 1.6; }
.state-current { color: var(--accent-primary-hover); }.state-waiting { color: var(--state-warning); }.state-failed { color: var(--state-danger); }.state-done { color: var(--state-online); }
.state-unknown, .state-pending, .state-cancelled, time { color: var(--text-secondary); }
.command, .legacy { display: grid; gap: var(--space-2); padding: var(--space-4); border: 1px solid var(--border-subtle); border-radius: var(--radius-md); }
.command.uncertain { border-color: color-mix(in srgb, var(--state-warning) 45%, var(--border-subtle)); }
p { color: var(--text-secondary); font-size: var(--text-sm); line-height: 1.65; overflow-wrap: anywhere; }
time, .evidence-meta, .evidence-heading span, .milestones { font-size: var(--text-xs); }
.evidence-heading { display: flex; flex-wrap: wrap; align-items: baseline; justify-content: space-between; gap: var(--space-2); padding-top: var(--space-2); }
.evidence-heading span { color: var(--text-secondary); }
.checks { border: 1px solid var(--border-subtle); border-radius: var(--radius-md); background: var(--surface); overflow: hidden; }
.check { padding: var(--space-5); display: grid; gap: var(--space-3); }.check + .check { border-top: 1px solid var(--border-subtle); }
.check-heading { display: flex; gap: var(--space-3); align-items: center; justify-content: space-between; flex-wrap: wrap; }.check-heading strong { font-size: var(--text-sm); }
dl { display: flex; gap: var(--space-4) var(--space-8); margin: 0; flex-wrap: wrap; }dt { font-size: var(--text-xs); color: var(--text-tertiary); }dd { margin: var(--space-1) 0 0; font-size: var(--text-sm); overflow-wrap: anywhere; }
.evidence-meta { display: flex; gap: var(--space-1) var(--space-4); flex-wrap: wrap; color: var(--text-tertiary); line-height: 1.6; }
.milestones { border-block: 1px solid var(--border-subtle); padding: var(--space-3) 0; }.milestones summary { color: var(--text-secondary); }.milestones ul { display: grid; gap: var(--space-3); padding-top: var(--space-4); }.milestones li { display: grid; gap: var(--space-1); }.milestones strong { font-weight: var(--weight-medium); }
.map-note { font-size: var(--text-xs); }code { font-family: var(--font-mono); overflow-wrap: anywhere; }
@media (max-width: 600px) { .phases summary { padding: var(--space-4); gap: var(--space-2); flex-wrap: wrap; }.phase-title { flex-basis: calc(100% - 70px); }.phase-state { padding-left: 34px; order: 1; flex-basis: 100%; }.phases summary::after { order: 0; }.steps, .check { padding: var(--space-4); } }
</style>
+202
View File
@@ -0,0 +1,202 @@
import { provisioningPhases, type ProvisioningDetail, type ProvisioningEvent, type ProvisioningEventPage, type ProvisioningPhase } from '$lib/provisioning-detail.js';
import { actionCompleted, actionLabels, provisioningActions, type ProvisioningAction } from '$lib/provisioning.js';
export const activeProvisioningStates = ['queued', 'waiting', 'running', 'awaiting-health'];
export type ProgressState = 'done' | 'current' | 'waiting' | 'failed' | 'pending' | 'cancelled' | 'unknown';
export interface ProgressStep { key: ProvisioningPhase; label: string; state: ProgressState; at: string | null }
export interface ProgressGroup { key: string; label: string; steps: ProgressStep[]; state: ProgressState }
const phaseLabels: Record<ProvisioningPhase, string> = {
requested: 'Request recorded', eligibility: 'Check eligibility and capacity', 'startup-queue': 'Wait for a startup slot',
prepare: 'Prepare server files', 'create-deployment': 'Create Komodo deployment', 'container-action': 'Start container',
'remove-container': 'Stop and remove container', 'delete-deployment': 'Delete Komodo deployment', cleanup: 'Remove files and registration',
'verify-container': 'Confirm container readiness', 'verify-game': 'Verify game status', 'verify-map': 'Verify selected engine map',
'verify-stopped': 'Confirm container is stopped'
};
type GroupDefinition = { key: string; label: string; phases: ProvisioningPhase[] };
const request: GroupDefinition = { key: 'request', label: 'Request and eligibility', phases: ['requested', 'eligibility'] };
const creation: GroupDefinition = { key: 'create', label: 'Create and start', phases: ['startup-queue', 'prepare', 'create-deployment', 'container-action'] };
const removal: GroupDefinition = { key: 'remove', label: 'Remove the instance', phases: ['remove-container', 'delete-deployment', 'cleanup'] };
const readiness: GroupDefinition = { key: 'verify', label: 'Verify readiness', phases: ['verify-container', 'verify-game', 'verify-map'] };
function definitions(action: ProvisioningAction): GroupDefinition[] {
if (action === 'create') return [{ ...request, phases: ['requested'] }, creation, readiness];
if (action === 'rebuild') return [request, removal, { ...creation, label: 'Create a fresh instance' }, readiness];
if (action === 'destroy') return [request, removal];
return [request, { key: 'command', label: `${actionLabels[action]} the existing container`, phases: ['container-action'] },
{ ...readiness, phases: action === 'stop' ? ['verify-stopped'] : ['verify-container', 'verify-game'] }];
}
export function provisioningGroups(detail: ProvisioningDetail): ProgressGroup[] {
const progress = detail.progress;
if (!progress) return [];
const groups = definitions(detail.deployment.action);
const phases = groups.flatMap(group => group.phases);
const current = progress.phase ? phases.indexOf(progress.phase) : -1;
const active = activeProvisioningStates.includes(detail.deployment.state);
return groups.map(group => {
const steps = group.phases.map((key): ProgressStep => {
const checkpoint = progress.checkpoints.find(item => item.key === key);
const checkKey = key === 'verify-container' ? 'container' : key === 'verify-game' ? 'game' : key === 'verify-map' ? 'map' : key === 'verify-stopped' ? 'stopped' : null;
const check = checkKey ? progress.checks.find(item => item.key === checkKey) : null;
const index = phases.indexOf(key);
const isCurrent = key === progress.phase;
const state: ProgressState = check ? ({ pass: 'done', fail: 'failed', waiting: 'waiting', unknown: 'unknown' } as const)[check.result]
: checkpoint ? 'done'
: isCurrent && ['failed', 'interrupted'].includes(detail.deployment.state) ? 'failed'
: isCurrent && active ? progress.blocker || ['waiting', 'queued'].includes(detail.deployment.state) ? 'waiting' : 'current'
: current >= 0 && index >= current && detail.deployment.state === 'cancelled' ? 'cancelled'
: active && current >= 0 && index > current ? 'pending' : 'unknown';
const label = key === 'container-action' && ['start', 'restart', 'stop'].includes(detail.deployment.action)
? `${actionLabels[detail.deployment.action]} existing container` : phaseLabels[key];
return { key, label, state, at: check?.checkedAt ?? checkpoint?.at ?? (isCurrent ? progress.phaseStartedAt : null) };
});
const states = steps.map(step => step.state);
const state = (['failed', 'current', 'waiting'] as const).find(value => states.includes(value))
?? (states.every(value => value === 'done') ? 'done' : states.includes('unknown') ? 'unknown' : states.includes('cancelled') ? 'cancelled' : 'pending');
return { key: group.key, label: group.label, steps, state };
});
}
export const progressStateLabels: Record<ProgressState, string> = {
done: 'Completed', current: 'In progress', waiting: 'Waiting', failed: 'Needs attention', pending: 'Not started', cancelled: 'Not run', unknown: 'Not recorded'
};
export const progressIcons: Record<ProgressState, string> = { done: '✓', current: '•', waiting: '◷', failed: '!', pending: '○', cancelled: '–', unknown: '?' };
export function provisioningOutcome(detail: ProvisioningDetail) {
const { deployment: operation, progress } = detail;
const active = activeProvisioningStates.includes(operation.state);
const failure = ['failed', 'interrupted'].includes(operation.state);
const headline = operation.state === 'done' ? actionCompleted[operation.action]
: operation.state === 'cancelled' ? `${actionLabels[operation.action]} request cancelled`
: failure ? `${actionLabels[operation.action]} ${operation.state === 'interrupted' ? 'interrupted' : 'failed'}`
: operation.state === 'warning' ? `${actionLabels[operation.action]} completed with warnings`
: progress?.blocker?.message ?? operation.stage;
let description: string;
if (detail.canResumeCleanup) description = 'The remote deployment was deleted. Resume the remaining local cleanup without repeating container removal.';
else if (detail.canResumeRebuild) description = 'The remote deployment was deleted. Resume local cleanup, then create a fresh instance in the reserved server slot.';
else if (detail.canResumeVerification) description = 'Resume observation of the existing instance. This does not create, restart, or rebuild a container.';
else if (operation.state === 'cancelled') description = 'This request is cancelled. Recorded prior work is preserved below; remaining work will not run.';
else if (failure) description = progress?.command?.state === 'uncertain'
? 'The remote response was uncertain. Inspect the recorded result and server before requesting another operation.'
: 'Review the recorded failure and execution evidence before submitting another operation.';
else if (operation.state === 'warning') description = 'The operation finished. Review the recorded warnings below for any remaining issue.';
else if (operation.state === 'done') description = !progress
? 'This operation was recorded as completed. Detailed verification evidence was not recorded.'
: operation.action === 'destroy' ? 'The instance removal completed. This record preserves its results after deletion.'
: operation.action === 'stop' ? 'The container was confirmed stopped. Its files and registration were retained.'
: 'The operation completed its readiness checks. These recorded results describe that attempt; current health is available on the server page.';
else if (active && progress?.phase === 'cleanup' && !progress.blocker) description = operation.action === 'rebuild'
? 'Remote deployment deletion was confirmed. Local cleanup must finish before a fresh instance is created in the reserved slot.'
: 'Remote deployment deletion was confirmed. The remaining local files and registration cleanup will continue without repeating container removal.';
else if (progress?.blocker) description = progress.blocker.automatic
? 'The worker will recheck this condition and continue automatically when eligible.'
: 'Review the blocking condition below before this operation can continue.';
else if (operation.state === 'queued') description = 'The request is saved. It will start when its dependencies and startup capacity allow.';
else if (operation.state === 'waiting') description = 'Waiting for the worker’s eligibility checks. The recorded stage below describes the last known condition.';
else description = operation.action === 'destroy' ? 'Removal is in progress. Confirmed checkpoints and remaining cleanup are recorded below.'
: operation.action === 'rebuild' ? 'The old instance is removed before a fresh instance is created in its reserved slot. Follow each checkpoint below.'
: 'The worker is executing this request and checking its resulting state.';
return { active, headline, description };
}
/** Accept return context only from the local operation surfaces we expose. */
export function provisioningReturn(value: string | null | undefined): { href: string; label: string } {
const fallback = { href: '/provisioning', label: 'Back to Provisioning' };
if (!value || !value.startsWith('/') || value.startsWith('//') || value.includes('\\')) return fallback;
try {
const url = new URL(value, 'https://electric-cherry.invalid');
if (url.origin !== 'https://electric-cherry.invalid') return fallback;
if (url.pathname === '/provisioning') {
const params = new URLSearchParams();
for (const key of ['view', 'action', 'status', 'server', 'limit']) {
const item = url.searchParams.get(key);
if (item) params.set(key, item);
}
return { href: '/provisioning' + (params.size ? `?${params}` : ''), label: 'Back to Provisioning' };
}
const match = url.pathname.match(/^\/(campaigns|betas|maintenance|collections\/launches)\/[^/]+$/);
if (match) return { href: url.pathname, label: `Back to ${match[1] === 'collections/launches' ? 'collection' : match[1] === 'campaigns' ? 'campaign' : match[1] === 'betas' ? 'beta' : 'maintenance'}` };
} catch { /* Fall back to the local list. */ }
return fallback;
}
export function elapsedTime(start: string | null | undefined, end: number): string {
if (!start || !Number.isFinite(Date.parse(start))) return 'Not recorded';
const seconds = Math.max(0, Math.floor((end - Date.parse(start)) / 1000));
if (seconds < 60) return `${seconds}s`;
if (seconds < 3600) return `${Math.floor(seconds / 60)}m ${seconds % 60}s`;
return `${Math.floor(seconds / 3600)}h ${Math.floor(seconds % 3600 / 60)}m`;
}
const record = (value: unknown): value is Record<string, unknown> => !!value && typeof value === 'object' && !Array.isArray(value);
const text = (value: unknown): value is string => typeof value === 'string';
const number = (value: unknown): value is number => typeof value === 'number' && Number.isFinite(value);
const nullableText = (value: unknown) => value === null || text(value);
const timestamp = (value: unknown): value is string => text(value) && Number.isFinite(Date.parse(value));
const nullableTime = (value: unknown) => value === null || timestamp(value);
const scalar = (value: unknown) => value === null || text(value) || number(value);
const localLink = (value: unknown) => value === null || (text(value) && value.startsWith('/') && !value.startsWith('//') && !value.includes('\\'));
const optionalLink = (value: unknown) => value === undefined || localLink(value);
const phase = (value: unknown) => value === null || provisioningPhases.includes(value as ProvisioningPhase);
const event = (value: unknown): value is ProvisioningEvent => record(value) && number(value.id) && timestamp(value.created_at) && text(value.message);
export function isProvisioningEventPage(value: unknown): value is ProvisioningEventPage {
return record(value) && Array.isArray(value.events) && value.events.every(event)
&& number(value.total) && number(value.totalEvents) && (value.latestId === null || number(value.latestId))
&& number(value.watermark) && (value.nextCursor === null || number(value.nextCursor));
}
/** Validate every nested shape consumed by the page before replacing its last good snapshot. */
export function isProvisioningDetail(value: unknown, id: string): value is ProvisioningDetail {
if (!record(value) || !record(value.deployment)) return false;
const operation = value.deployment;
if (operation.id !== id || !provisioningActions.includes(operation.action as ProvisioningAction)
|| !['server', 'state', 'stage', 'triggered_by', 'warnings'].every(key => text(operation[key]))
|| !timestamp(operation.started_at) || !nullableTime(operation.finished_at) || !nullableTime(operation.server_deleted_at)
|| !nullableText(operation.error) || !number(operation.server_exists)
|| (operation.submitted_at !== undefined && !nullableTime(operation.submitted_at))
|| (operation.expected_map !== undefined && !nullableText(operation.expected_map))) return false;
if (!timestamp(value.generatedAt) || !Array.isArray(value.events) || !value.events.every(event)
|| !Array.isArray(value.warningMessages) || !value.warningMessages.every(text)
|| !['canResumeVerification', 'canResumeCleanup', 'canResumeRebuild', 'canCancelRemoval'].every(key => typeof value[key] === 'boolean')
|| !nullableText(value.blueprintName)) return false;
if (!record(value.activity) || !number(value.activity.total) || !(value.activity.latestId === null || number(value.activity.latestId))
|| !(value.activity.failureExcerpt === null || event(value.activity.failureExcerpt))) return false;
if (!Array.isArray(value.sources) || !value.sources.every(source => record(source)
&& ['campaign', 'beta', 'maintenance', 'watchdog'].includes(source.kind as string)
&& text(source.source_id) && text(source.label) && localLink(source.href))) return false;
if (!(value.collection === null || record(value.collection) && text(value.collection.id) && text(value.collection.name))) return false;
if (!record(value.capacity) || !timestamp(value.capacity.evaluatedAt) || !Array.isArray(value.capacity.pools)
|| !value.capacity.pools.every(pool => record(pool) && ['startup', 'restart'].includes(pool.kind as string)
&& number(pool.limit) && number(pool.used) && typeof pool.ownsSlot === 'boolean' && (pool.requestOrder === null || number(pool.requestOrder))
&& Array.isArray(pool.holders) && pool.holders.every(holder => record(holder) && nullableText(holder.id) && text(holder.server)))) return false;
if (value.recipe !== null) {
const recipe = value.recipe;
if (!record(recipe) || !['t5', 't6'].includes(recipe.game as string) || !timestamp(recipe.capturedAt)
|| !['mapSlug', 'mapName', 'expectedMap', 'image', 'configFile', 'configSource', 'fingerprint'].every(key => text(recipe[key]))
|| !Array.isArray(recipe.files) || !recipe.files.every(file => record(file)
&& ['name', 'kind', 'path', 'sha256'].every(key => text(file[key])) && number(file.size) && (file.version === null || number(file.version)))) return false;
}
if (value.progress !== null) {
const progress = value.progress;
if (!record(progress) || progress.version !== 1 || !phase(progress.phase) || !nullableTime(progress.phaseStartedAt) || !nullableTime(progress.observedAt)
|| ![null, 'queued', 'attempted'].includes(progress.recipeStatus as null | string)) return false;
if (!(progress.blocker === null || record(progress.blocker) && text(progress.blocker.code) && text(progress.blocker.message)
&& typeof progress.blocker.automatic === 'boolean' && optionalLink(progress.blocker.href)
&& (progress.blocker.label === undefined || nullableText(progress.blocker.label)))) return false;
if (!Array.isArray(progress.checks) || !progress.checks.every(check => record(check)
&& ['container', 'game', 'map', 'occupancy', 'stopped'].includes(check.key as string)
&& ['pass', 'waiting', 'fail', 'unknown'].includes(check.result as string)
&& [null, 'komodo', 'rcon', 'telemetry', 'combined'].includes(check.source as null | string)
&& text(check.label) && timestamp(check.checkedAt) && nullableTime(check.receivedAt)
&& scalar(check.expected) && scalar(check.actual) && nullableText(check.message))) return false;
if (!Array.isArray(progress.checkpoints) || !progress.checkpoints.every(checkpoint => record(checkpoint)
&& text(checkpoint.key) && text(checkpoint.label) && timestamp(checkpoint.at) && phase(checkpoint.phase))) return false;
if (!(progress.command === null || record(progress.command) && ['attempted', 'accepted', 'rejected', 'uncertain'].includes(progress.command.state as string)
&& timestamp(progress.command.at) && text(progress.command.message))) return false;
if (!(progress.recovery === null || record(progress.recovery) && ['held', 'released', 'transferred', 'unknown'].includes(progress.recovery.state as string)
&& timestamp(progress.recovery.at) && text(progress.recovery.message)
&& (progress.recovery.operationId === undefined || nullableText(progress.recovery.operationId)))) return false;
}
return true;
}
+131
View File
@@ -0,0 +1,131 @@
export type ProvisioningMutation = 'resume' | 'cancel';
export interface ProvisioningRefreshState<T> {
id: string;
detail: T | null;
refreshing: boolean;
busy: ProvisioningMutation | null;
loadError: string;
actionError: string;
notice: string;
refreshedAt: string | null;
}
interface Options<T> {
onChange: (state: ProvisioningRefreshState<T>) => void;
validate: (value: unknown, id: string) => value is T;
fetcher?: typeof fetch;
now?: () => string;
timeoutMs?: number;
}
/** Reads and action feedback belong to a particular visit, even when its ID is revisited. */
export function createProvisioningRefresh<T>(options: Options<T>) {
const request = options.fetcher ?? fetch;
const now = options.now ?? (() => new Date().toISOString());
let visit = 0;
let read = 0;
let disposed = false;
let pending: { controller: AbortController; promise: Promise<void>; timer?: ReturnType<typeof setTimeout> } | null = null;
let mutation: AbortController | null = null;
let state: ProvisioningRefreshState<T> = {
id: '', detail: null, refreshing: false, busy: null, loadError: '', actionError: '', notice: '', refreshedAt: null
};
const publish = (patch: Partial<ProvisioningRefreshState<T>>) => {
state = { ...state, ...patch };
if (!disposed) options.onChange(state);
};
function invalidateRead() {
read++;
if (pending) {
clearTimeout(pending.timer);
pending.controller.abort();
pending = null;
}
}
async function load(force = false, reconcile = false): Promise<void> {
if (disposed || !state.id || (state.busy && !reconcile)) return;
if (pending && !force) return pending.promise;
invalidateRead();
const currentVisit = visit;
const currentRead = read;
const id = state.id;
const current = () => !disposed && currentVisit === visit && currentRead === read && id === state.id;
const controller = new AbortController();
const entry = { controller, promise: Promise.resolve(), timer: undefined as ReturnType<typeof setTimeout> | undefined };
let timedOut = false;
pending = entry;
publish({ refreshing: true });
entry.timer = setTimeout(() => { timedOut = true; controller.abort(); }, options.timeoutMs ?? 15_000);
entry.promise = (async () => {
try {
const response = await request(`/api/provisioning/${encodeURIComponent(id)}?events=recent`, { signal: controller.signal, cache: 'no-store' });
if (!response.ok) throw new Error(response.status === 404 ? 'Provisioning operation not found.' : 'Could not refresh this operation.');
const detail: unknown = await response.json();
if (!options.validate(detail, id)) throw new Error('The operation response was incomplete. Try refreshing again.');
if (current()) publish({ detail, loadError: '', refreshedAt: now() });
} catch (error) {
if (current() && (!controller.signal.aborted || timedOut)) publish({
loadError: timedOut ? 'Operation refresh timed out. Try again.' : error instanceof Error ? error.message : 'Could not refresh this operation.'
});
} finally {
clearTimeout(entry.timer);
if (pending === entry) pending = null;
if (current()) publish({ refreshing: false });
}
})();
return entry.promise;
}
function setOperation(id: string, detail: T | null = null, refreshedAt: string | null = null) {
if (disposed) return;
visit++;
invalidateRead();
mutation?.abort();
mutation = null;
publish({ id, detail, refreshedAt, refreshing: false, busy: null, loadError: '', actionError: '', notice: '' });
if (!detail) void load();
}
async function act(action: ProvisioningMutation, successMessage: string): Promise<void> {
if (disposed || !state.id || state.busy) return;
const currentVisit = visit;
const id = state.id;
const current = () => !disposed && currentVisit === visit && id === state.id;
invalidateRead();
const controller = new AbortController();
mutation = controller;
let timedOut = false;
const timer = setTimeout(() => { timedOut = true; controller.abort(); }, options.timeoutMs ?? 15_000);
publish({ busy: action, refreshing: false, actionError: '', notice: '' });
try {
const response = await request(`/api/provisioning/${encodeURIComponent(id)}/${action}`, { method: 'POST', signal: controller.signal });
const body = await response.json() as { error?: string };
if (!response.ok) throw new Error(body.error ?? 'The operation changed before this request could be accepted.');
if (current()) publish({ notice: successMessage });
} catch (error) {
if (current() && (!controller.signal.aborted || timedOut)) publish({ actionError: timedOut
? 'The request timed out and may have reached the server. Review the refreshed operation before trying again.'
: error instanceof Error ? error.message : 'Could not confirm the request. Review the operation before trying again.' });
} finally {
clearTimeout(timer);
if (mutation === controller) mutation = null;
// A failed reconciliation is a read error, never a failed accepted mutation.
if (current()) {
await load(true, true);
if (current()) publish({ busy: null });
}
}
}
function dispose() {
disposed = true;
visit++;
invalidateRead();
mutation?.abort();
mutation = null;
}
return { setOperation, refresh: () => load(), forceRefresh: () => load(true), act, dispose };
}
+2 -2
View File
@@ -2,8 +2,8 @@ import { json, error } from '@sveltejs/kit';
import { db } from '$lib/server/db/index.js'; import { db } from '$lib/server/db/index.js';
import { getDeployment } from '$lib/server/provisioning/index.js'; import { getDeployment } from '$lib/server/provisioning/index.js';
export async function GET({ params }: { params: { id: string } }) { export async function GET({ params, url }: { params: { id: string }; url?: URL }) {
const lifecycle = getDeployment(params.id); const lifecycle = getDeployment(params.id, { events: url?.searchParams.get('events') === 'recent' ? 'recent' : 'all' });
if (lifecycle) return json(lifecycle); if (lifecycle) return json(lifecycle);
const dep = db() const dep = db()
.prepare('SELECT * FROM deployments WHERE id = ?') .prepare('SELECT * FROM deployments WHERE id = ?')
@@ -0,0 +1,24 @@
import { json, error } from '@sveltejs/kit';
import { db } from '$lib/server/db/index.js';
import { deploymentActivity } from '$lib/server/provisioning/index.js';
function integer(params: URLSearchParams, key: string): number | undefined {
const raw = params.get(key);
if (raw === null) return undefined;
if (!/^\d+$/.test(raw)) error(400, `${key} must be a nonnegative integer.`);
const value = Number(raw);
if (!Number.isSafeInteger(value)) error(400, `${key} must be a nonnegative integer.`);
return value;
}
export function GET({ params, url }: { params: { id: string }; url: URL }) {
if (!db().prepare('SELECT 1 FROM server_deployments WHERE id = ?').get(params.id)) error(404, 'Provisioning operation not found.');
const order = url.searchParams.get('order') ?? 'newest';
if (order !== 'newest' && order !== 'oldest') error(400, 'Choose newest or oldest activity order.');
const query = url.searchParams.get('query') ?? '';
if (query.length > 500) error(400, 'Activity search must be 500 characters or fewer.');
const limit = integer(url.searchParams, 'limit');
if (limit === 0) error(400, 'Activity limit must be at least 1.');
return json(deploymentActivity(params.id, { order, query, limit,
cursor: integer(url.searchParams, 'cursor'), watermark: integer(url.searchParams, 'watermark') }));
}
+1 -1
View File
@@ -128,7 +128,7 @@
</div> </div>
{#snippet row(d: Deployment)} {#snippet row(d: Deployment)}
<article class="deployment-row"><a class="row-link" href="/provisioning/{d.id}"> <article class="deployment-row"><a class="row-link" href={`/provisioning/${encodeURIComponent(d.id)}?returnTo=${encodeURIComponent($page.url.pathname + $page.url.search)}`}>
<div class="row-head"> <div class="row-head">
<div><Badge variant={d.action === 'create' ? 'info' : 'neutral'}>{actionLabels[d.action]}</Badge> <div><Badge variant={d.action === 'create' ? 'info' : 'neutral'}>{actionLabels[d.action]}</Badge>
<strong class="server"><HistoricalServerIdentity name={d.server} deletedAt={d.server_deleted_at} /></strong> <strong class="server"><HistoricalServerIdentity name={d.server} deletedAt={d.server_deleted_at} /></strong>
+2 -2
View File
@@ -2,8 +2,8 @@ import { redirect, error } from '@sveltejs/kit';
import { getDeployment } from '$lib/server/provisioning/index.js'; import { getDeployment } from '$lib/server/provisioning/index.js';
import { db } from '$lib/server/db/index.js'; import { db } from '$lib/server/db/index.js';
export function load({ params }: { params: { id: string } }) { export function load({ params }: { params: { id: string } }) {
const detail = getDeployment(params.id); const detail = getDeployment(params.id, { events: 'recent' });
if (detail) return {}; if (detail) return { detail };
if (db().prepare('SELECT id FROM deployments WHERE id = ?').get(params.id)) { if (db().prepare('SELECT id FROM deployments WHERE id = ?').get(params.id)) {
redirect(307, '/campaigns/' + encodeURIComponent(params.id)); redirect(307, '/campaigns/' + encodeURIComponent(params.id));
} }
+156 -155
View File
@@ -1,177 +1,178 @@
<script lang="ts"> <script lang="ts">
import { onDestroy, onMount, untrack } from 'svelte';
import { page } from '$app/stores';
import type { PageData } from './$types';
import type { ProvisioningDetail } from '$lib/provisioning-detail.js';
import { actionLabels } from '$lib/provisioning.js';
import { timeFormat } from '$ui/stores/time-zone.js'; import { timeFormat } from '$ui/stores/time-zone.js';
import { onMount } from 'svelte';
import { pollVisible } from '$ui/poll.js'; import { pollVisible } from '$ui/poll.js';
import { serverTitle } from '$ui/server-labels.js'; import { serverTitle } from '$ui/server-labels.js';
import StatusBadge from '$ui/components/primitives/StatusBadge.svelte'; import { toast } from '$ui/toast.js';
import OperationHistory from '$ui/components/OperationHistory.svelte'; import { createProvisioningRefresh, type ProvisioningRefreshState } from '$ui/provisioning-refresh.js';
import type { OperationStep } from '$lib/operation-flow.js'; import { elapsedTime, isProvisioningDetail, provisioningOutcome, provisioningReturn } from '$ui/provisioning-presentation.js';
import { page } from '$app/stores';
import SectionHeader from '$ui/components/primitives/SectionHeader.svelte'; import SectionHeader from '$ui/components/primitives/SectionHeader.svelte';
import Breadcrumbs from '$ui/components/primitives/Breadcrumbs.svelte'; import Breadcrumbs from '$ui/components/primitives/Breadcrumbs.svelte';
import StatusBadge from '$ui/components/primitives/StatusBadge.svelte';
import Badge from '$ui/components/primitives/Badge.svelte'; import Badge from '$ui/components/primitives/Badge.svelte';
import Button from '$ui/components/primitives/Button.svelte'; import Button from '$ui/components/primitives/Button.svelte';
import DeployServerDialog from '$ui/components/DeployServerDialog.svelte'; import Card from '$ui/components/primitives/Card.svelte';
import CopyValue from '$ui/components/CopyValue.svelte';
import ProvisioningProgress from '$ui/components/ProvisioningProgress.svelte';
import ProvisioningActivity from '$ui/components/ProvisioningActivity.svelte';
import LaunchRecipe from '$ui/components/LaunchRecipe.svelte'; import LaunchRecipe from '$ui/components/LaunchRecipe.svelte';
import type { LaunchRecipeSummary } from '$lib/deployment-plan.js'; import DeployServerDialog from '$ui/components/DeployServerDialog.svelte';
import { actionLabels, actionCompleted, type ProvisioningAction } from '$lib/provisioning.js'; let { data }: { data: PageData } = $props();
const initial = untrack(() => data.detail);
interface Deployment { let view = $state<ProvisioningRefreshState<ProvisioningDetail>>({
id: string; expected_map?: string; server: string; server_deleted_at?: string | null; action: ProvisioningAction; state: string; id: initial.deployment.id, detail: initial, refreshing: false, busy: null,
stage: string; started_at: string; finished_at: string | null; loadError: '', actionError: '', notice: '', refreshedAt: initial.generatedAt
triggered_by: string; error: string | null; warnings: string; server_exists: number;
}
interface Detail { deployment: Deployment; events: { id: number; created_at: string; message: string }[]; sources: { kind: 'campaign' | 'beta' | 'maintenance' | 'watchdog'; source_id: string }[]; recipe: LaunchRecipeSummary | null; blueprintName: string | null; canResumeVerification: boolean; canResumeCleanup: boolean; canResumeRebuild: boolean; canCancelRemoval: boolean; collection: { id: string; name: string } | null }
let detail = $state<Detail | null>(null);
let error = $state('');
let refreshing = false;
let now = $state(Date.now());
let open = $state(false);
let resuming = $state(false);
let deployment = $derived(detail?.deployment);
let active = $derived(!deployment?.server_deleted_at && ['queued', 'awaiting-health', 'waiting', 'running'].includes(deployment?.state ?? ''));
let warnings = $derived.by(() => {
try { return JSON.parse(deployment?.warnings ?? '[]') as string[]; } catch { return []; }
}); });
const createSteps = ['Prepare server files', 'Create Komodo deployment', 'Start container', 'Verify Komodo health', 'Verify RCON health', 'Verify selected map']; let now = $state(Date.parse(initial.generatedAt));
const createStages = ['Preparing server files', 'Creating Komodo deployment', 'Starting container', 'Waiting for Komodo health', 'Waiting for RCON health', 'Waiting for selected map']; let launchOpen = $state(false);
const destroySteps = ['Wait for eligibility', 'Stop and remove container', 'Delete Komodo deployment', 'Remove server files and registry']; const controller = createProvisioningRefresh<ProvisioningDetail>({ validate: isProvisioningDetail, onChange: value => { view = value; } });
const destroyStages = ['Waiting to verify', 'Stopping and removing container', 'Deleting Komodo deployment', 'Removing server files and registry']; const detail = $derived(view.detail);
let steps = $derived(deployment?.action === 'destroy' ? ['Request destruction', ...destroySteps] const deployment = $derived(detail?.deployment);
: deployment?.action === 'rebuild' ? ['Request rebuild', ...destroySteps, 'Queue fresh instance', ...createSteps] const outcome = $derived(detail ? provisioningOutcome(detail) : null);
: deployment?.action === 'create' ? ['Request creation', ...createSteps] const returnTo = $derived.by(() => {
: ['Request operation', 'Wait for eligibility', `${deployment ? actionLabels[deployment.action] : 'Start'} container`, deployment?.action === 'stop' ? 'Verify container stopped' : 'Verify container and RCON health']); const requested = $page.url.searchParams.get('returnTo');
let currentStep = $derived.by(() => { if (requested) return provisioningReturn(requested);
if (!deployment) return 0; if (detail?.collection) return { href: `/collections/launches/${encodeURIComponent(detail.collection.id)}`, label: `Back to ${detail.collection.name}` };
if (deployment.state === 'done' || deployment.state === 'warning') return steps.length; if (detail?.sources.length === 1 && detail.sources[0].href) return { href: detail.sources[0].href, label: `Back to ${detail.sources[0].label}` };
const { action, state, stage } = deployment; return provisioningReturn(null);
if (state === 'waiting' && stage !== 'Removing server files and registry') return 1;
if (action === 'start' || action === 'restart' || action === 'stop') return state === 'awaiting-health' ? 3 : state === 'running' ? 2 : stage.includes('container') ? 2 : 1;
if (state === 'queued') return action === 'rebuild' ? 5 : 0;
const stages = action === 'destroy' ? ['Starting', ...destroyStages]
: action === 'rebuild' ? ['Starting', ...destroyStages, 'Queued for server creation', ...createStages]
: ['Starting', ...createStages];
const index = stages.findIndex(value => stage === value || stage.startsWith(value + ':'));
if (index < 0 && state === 'awaiting-health') return action === 'rebuild' ? 9 : 4;
return Math.max(0, index);
}); });
let explanation = $derived.by(() => { const serverHref = $derived(deployment && deployment.server_exists && !deployment.server_deleted_at ? `/servers/${encodeURIComponent(deployment.server)}` : null);
if (!deployment) return ''; const recipeStatus = $derived(detail?.progress?.recipeStatus ?? null);
if (deployment.server_deleted_at) return 'This server was deleted. Its recorded operation and results are preserved below.'; const recipeHeading = $derived(recipeStatus === 'queued' ? 'Setup captured when queued' : recipeStatus === 'attempted' ? 'Files selected for this attempt' : 'Recorded launch files');
if (detail?.canResumeRebuild) return 'The old Komodo instance was deleted. Resume file cleanup, then create a fresh instance in the reserved server slot.'; $effect(() => {
if (['start', 'restart', 'stop'].includes(deployment.action)) { const snapshot = data.detail;
if (deployment.state === 'done') return deployment.action === 'stop' ? 'Komodo confirmed the container is stopped. Start it again when ready.' : 'The container is healthy and RCON responds.'; untrack(() => {
if (['failed', 'interrupted'].includes(deployment.state)) return 'Inspect the server before submitting another operation. An uncertain response may have reached Komodo.'; launchOpen = false;
return deployment.state === 'waiting' ? 'Waiting for other work and a safe server state before continuing.' : 'The request is saved. Verifying the resulting container state automatically.'; now = Date.parse(snapshot.generatedAt);
} controller.setOperation(snapshot.deployment.id, snapshot, snapshot.generatedAt);
if (deployment.state === 'cancelled') return 'The queued removal was cancelled before removal began. The server and its files were kept.';
if (detail?.canResumeCleanup) return 'Komodo deletion succeeded. Resume the remaining file and database cleanup; the removed container does not need to answer RCON.';
if (deployment.state === 'waiting' && deployment.stage === 'Removing server files and registry') return 'Local cleanup is queued. Komodo deletion has already succeeded.';
if (deployment.state === 'done') return ['create', 'rebuild'].includes(deployment.action)
? 'The container is healthy, RCON responds, and the reported engine map matches your selection.'
: 'The container, deployment, and server registration have been removed.';
if (deployment.state === 'warning') return 'The operation finished with the warnings below. Review any remaining cleanup.';
if (deployment.state === 'failed' || deployment.state === 'interrupted') return detail?.canResumeVerification
? 'The server is still registered. Inspect its status, then resume health verification without creating or restarting a container.'
: 'Inspect the error and any remaining container or files before launching again. The activity below records how far this operation reached.';
if (deployment.state === 'queued') return 'Your request is saved in the launch queue. A startup slot opens after an earlier server passes its container, RCON, and map checks.';
if (deployment.action === 'rebuild' && deployment.state === 'waiting') return 'The rebuild waits for other work to finish, an empty server, and available recovery and startup capacity. The old instance and all of its files will be deleted before a fresh instance is created.';
if (deployment.state === 'waiting') return 'The server stays online while players finish. Removal begins only after a live check confirms it is empty.';
if (deployment.stage.startsWith('Waiting for selected map')) return 'Container and RCON checks passed. Waiting for RCON to report the selected engine map. Inspect the server configuration if this persists.';
if (deployment.stage.startsWith('Waiting for RCON')) return 'The container is healthy. Waiting for the game to answer RCON; check its startup logs, password, and network if this persists.';
if (deployment.state === 'awaiting-health') return 'Waiting for the container to become healthy. Verification runs automatically; startup logs can help explain a delay.';
return ['create', 'rebuild'].includes(deployment.action) ? 'Installing the latest promoted setup and starting its container. Health checks follow automatically.' : 'Removing the empty server and recording cleanup results.';
}); });
function duration(start: string, end: number) { });
const seconds = Math.max(0, Math.floor((end - new Date(start).getTime()) / 1000)); onMount(() => pollVisible(() => { now = Date.now(); return controller.refresh(); }, 5000, false));
if (seconds < 60) return `${seconds}s`; onDestroy(() => controller.dispose());
if (seconds < 3600) return `${Math.floor(seconds / 60)}m ${seconds % 60}s`; function fmt(value: string | null | undefined) { return value ? $timeFormat.dateTime(value) : 'Not recorded'; }
return `${Math.floor(seconds / 3600)}h ${Math.floor(seconds % 3600 / 60)}m`; async function copyLink() {
}
let stageStarted = $derived(detail?.events.slice().reverse().find(event => event.message === deployment?.stage)?.created_at ?? deployment?.started_at);
async function resume() {
if (!deployment || resuming) return;
resuming = true;
try { try {
const res = await fetch(`/api/provisioning/${encodeURIComponent(deployment.id)}/resume`, { method: 'POST' }); const url = new URL($page.url);
const body = await res.json(); url.search = ''; url.hash = '';
if (!res.ok) throw new Error(body.error ?? 'Could not resume the operation.'); await navigator.clipboard.writeText(url.toString());
await load(); toast('Operation link copied.', { variant: 'success' });
} catch (err) { error = (err as Error).message; } } catch { toast('Copy unavailable. Copy this page’s address from your browser.', { variant: 'warning' }); }
finally { resuming = false; }
} }
async function cancelRemoval() {
if (!deployment || resuming) return;
resuming = true; error = '';
try {
const response = await fetch(`/api/provisioning/${encodeURIComponent(deployment.id)}/cancel`, { method: 'POST' });
const body = await response.json();
if (!response.ok) throw new Error(body.error ?? 'Could not cancel this request.');
await load();
} catch (err) { error = (err as Error).message; await load(); error = (err as Error).message; }
finally { resuming = false; }
}
async function load() {
if (refreshing) return;
refreshing = true;
const id = $page.params.id ?? '';
try {
const res = await fetch('/api/provisioning/' + encodeURIComponent(id));
if (!res.ok) throw new Error(res.status === 404 ? 'Provisioning operation not found.' : 'Could not refresh provisioning operation.');
const body = await res.json();
if ($page.params.id === id) { detail = body; error = ''; }
} catch (err) { error = (err as Error).message; }
finally { refreshing = false; }
}
$effect(() => { $page.params.id; detail = null; void load(); });
onMount(() => pollVisible(() => { now = Date.now(); return load(); }, 5000, false));
function fmt(iso: string) { return $timeFormat.dateTime(iso); }
</script> </script>
<svelte:head><title>{deployment ? actionLabels[deployment.action] + ' ' + deployment.server : 'Provisioning'} | Electric Cherry</title></svelte:head> <svelte:head><title>{deployment ? `${actionLabels[deployment.action]} ${deployment.server}` : 'Provisioning'} | Electric Cherry</title></svelte:head>
<div class="page">
<SectionHeader title={deployment ? actionLabels[deployment.action] + ' server' : 'Provisioning'} subtitle={deployment ? serverTitle(deployment.server) : undefined}>
{#snippet crumbs()}<Breadcrumbs items={[{ label: 'Provisioning', href: '/provisioning' }, { label: deployment ? serverTitle(deployment.server) : 'Details' }]} />{/snippet}
</SectionHeader>
{#if error}<p role="alert" class="error">{error} <button onclick={load}>Retry</button></p>{/if}
{#if detail?.collection}<a href="/collections/launches/{detail.collection.id}">← Back to {detail.collection.name} collection</a>{/if}
{#if deployment}
<section class="summary">
<div class="top"><StatusBadge state={deployment.state} />
{#if deployment.server_deleted_at}<Badge>Deleted</Badge>{:else if deployment.server_exists}<a href="/servers/{encodeURIComponent(deployment.server)}">View server</a>{/if}
</div>
<h2>{deployment.state === 'done' ? actionCompleted[deployment.action] : deployment.stage}</h2>
<p class="explanation" aria-live="polite">{explanation}</p>
<div class="timing"><span>Total time <strong>{duration(deployment.started_at, deployment.finished_at ? new Date(deployment.finished_at).getTime() : now)}</strong></span>{#if active && stageStarted}<span>This step <strong>{duration(stageStarted, now)}</strong></span>{/if}</div>
{#if detail?.canResumeVerification && !deployment.server_deleted_at}<div><Button disabled={resuming} loading={resuming} onclick={resume}>Resume health verification</Button></div>{/if}
{#if detail?.canResumeRebuild && !deployment.server_deleted_at}<div><Button disabled={resuming} loading={resuming} onclick={resume}>Resume rebuild cleanup</Button></div>{/if}
{#if detail?.canResumeCleanup && !deployment.server_deleted_at}<div><Button disabled={resuming} loading={resuming} onclick={resume}>Resume cleanup</Button></div>{/if}
{#if detail?.canCancelRemoval && !deployment.server_deleted_at}<div><Button disabled={resuming} loading={resuming} onclick={cancelRemoval}>Cancel queued removal</Button><p class="muted">Available until removal begins. Cancelling keeps the server and its files.</p></div>{/if}
{#if deployment.server_exists && !deployment.server_deleted_at && deployment.state !== 'done' && deployment.state !== 'cancelled'}<div class="recovery-links"><a href="/servers/{encodeURIComponent(deployment.server)}?tab=logs">Open startup logs</a><a href="/servers/{encodeURIComponent(deployment.server)}?tab=diagnostics">Diagnose connection</a>{#if deployment.stage.includes('map')}<a href="/servers/{encodeURIComponent(deployment.server)}?tab=config">Review map configuration</a>{/if}</div>{/if}
{#if deployment.error}<p role="alert" class="error">{deployment.error}</p>{/if}
{#each warnings as warning}<p class="warning">{warning}</p>{/each}
{#if deployment.expected_map}<p>Expected map: <strong>{deployment.expected_map}</strong></p>{/if}
<dl><div><dt>Started</dt><dd>{fmt(deployment.started_at)}</dd></div><div><dt>Finished</dt><dd>{deployment.finished_at ? fmt(deployment.finished_at) : 'In progress'}</dd></div><div><dt>Requested by</dt><dd>{deployment.triggered_by}</dd></div></dl>
{#if detail?.sources.length}<div class="recovery-links" aria-label="Requesting work">{#each detail.sources as source}<a href={source.kind === 'campaign' ? `/campaigns/${encodeURIComponent(source.source_id)}` : source.kind === 'beta' ? `/betas/${encodeURIComponent(source.source_id)}` : source.kind === 'watchdog' ? `/maintenance/${encodeURIComponent(source.source_id)}` : `/maintenance?view=activity#job-${source.source_id}`}>{source.kind === 'campaign' ? 'Campaign' : source.kind === 'beta' ? 'Beta' : source.kind === 'watchdog' ? 'Maintenance finding' : 'Maintenance'} {source.source_id}</a>{/each}</div>{/if}
</section>
{#if detail?.recipe}<section class="saved-recipe"><div class="top"><h2>{detail.blueprintName ? `Launched from ${detail.blueprintName}` : 'Recorded launch files'}</h2><Button variant="ghost" onclick={() => open = true}>Launch another server →</Button></div><p class="muted">Launch this map with the latest promoted files using a fresh server ID and port. This does not repair the existing server.</p><LaunchRecipe recipe={detail.recipe} historical /></section><DeployServerDialog bind:open sourceDeploymentId={deployment.id} />{/if}
<OperationHistory steps={steps.map((label,index): OperationStep => ({label,state:deployment.state === 'cancelled' ? 'pending' : index < currentStep ? 'done' : index === currentStep ? (!active && ['failed','interrupted'].includes(deployment.state) ? 'failed' : 'current') : 'pending'}))} events={detail?.events ?? []} {active} />
{:else if !error}<p>Loading provisioning operation...</p>{/if} <div class="page">
<SectionHeader title={deployment ? `${actionLabels[deployment.action]} · ${serverTitle(deployment.server)}` : 'Provisioning operation'} subtitle={deployment?.server}>
{#snippet crumbs()}<Breadcrumbs items={[{ label: 'Provisioning', href: returnTo.href.startsWith('/provisioning') ? returnTo.href : '/provisioning' }, { label: 'Operation' }]} />{/snippet}
{#snippet actions()}{#if serverHref}<Button href={serverHref}>View server</Button>{/if}{/snippet}
</SectionHeader>
<div class="identity-row">
{#if deployment}<div class="operation-id"><span>Operation</span><CopyValue value={deployment.id} label="operation ID" /></div>{/if}
<div class="identity-links"><a href={returnTo.href}>← {returnTo.label}</a><Button size="sm" variant="ghost" onclick={copyLink}>Copy link</Button></div>
</div>
{#if view.loadError}<div class="refresh-error" role="alert"><div><strong>Refresh unavailable</strong><p>{view.loadError} {detail ? 'Showing the last successfully loaded record.' : ''}</p></div><Button size="sm" disabled={view.refreshing || !!view.busy} onclick={() => controller.forceRefresh()}>Retry refresh</Button></div>{/if}
{#if deployment && detail && outcome}
<section class="outcome" class:attention={['failed', 'interrupted', 'warning'].includes(deployment.state)} aria-labelledby="outcome-title">
<div class="outcome-top"><div class="badges"><StatusBadge state={deployment.state} />{#if deployment.server_deleted_at}<Badge>Deleted identity</Badge>{/if}{#if detail.progress?.recovery?.state === 'held'}<Badge variant="warning">Recovery capacity held</Badge>{/if}</div><span class="record-label">Operation outcome</span></div>
<div class="outcome-message" aria-live="polite" aria-atomic="true"><h2 id="outcome-title">{outcome.headline}</h2><p>{outcome.description}</p></div>
{#if deployment.error}<div class="failure" role="alert"><strong>Recorded error</strong><p>{deployment.error}</p></div>{/if}
{#if detail.activity.failureExcerpt && detail.activity.failureExcerpt.message !== deployment.error && ['failed', 'interrupted'].includes(deployment.state)}<p class="failure-excerpt"><span>Last failure event · {fmt(detail.activity.failureExcerpt.created_at)}</span>{detail.activity.failureExcerpt.message}</p>{/if}
{#if detail.warningMessages.length}<div class="warnings"><h3>Recorded warnings</h3><ul>{#each detail.warningMessages as warning}<li>{warning}</li>{/each}</ul></div>{/if}
{#if detail.canResumeVerification || detail.canResumeCleanup || detail.canResumeRebuild || detail.canCancelRemoval}
<div class="recovery-actions">
{#if detail.canResumeVerification}<Button variant="primary" loading={view.busy === 'resume'} disabled={!!view.busy} onclick={() => controller.act('resume', 'Health verification requested. The existing instance will be observed without another creation or restart.')}>Resume health verification</Button>{/if}
{#if detail.canResumeRebuild}<Button variant="primary" loading={view.busy === 'resume'} disabled={!!view.busy} onclick={() => controller.act('resume', 'Rebuild cleanup requested. Fresh creation waits for local cleanup to finish.')}>Resume rebuild cleanup</Button>{/if}
{#if detail.canResumeCleanup}<Button variant="primary" loading={view.busy === 'resume'} disabled={!!view.busy} onclick={() => controller.act('resume', 'Local cleanup requested. Remote container deletion will not be repeated.')}>Resume cleanup</Button>{/if}
{#if detail.canCancelRemoval}<div><Button disabled={!!view.busy} loading={view.busy === 'cancel'} onclick={() => controller.act('cancel', 'Queued removal cancelled. The server and its files were kept.')}>Cancel queued removal</Button><p class="action-help">Available until removal begins. Cancelling keeps the server and its files.</p></div>{/if}
</div>
{/if}
{#if view.actionError}<div class="action-error" role="alert"><strong>Request not confirmed</strong><p>{view.actionError}</p></div>{/if}
{#if view.notice}<p class="notice" role="status">{view.notice}</p>{/if}
{#if detail.progress?.blocker?.href}<a class="dependency" href={detail.progress.blocker.href}>{detail.progress.blocker.label ?? 'View blocking work'} →</a>{/if}
{#if detail.progress?.recovery}<div class="recovery-note"><strong>{detail.progress.recovery.state === 'held' ? 'Recovery remains in progress' : detail.progress.recovery.state === 'released' ? 'Recovery capacity released' : detail.progress.recovery.state === 'transferred' ? 'Recovery transferred' : 'Recovery state not confirmed'}</strong><p>{detail.progress.recovery.message}</p><span>Recorded {fmt(detail.progress.recovery.at)}</span>{#if detail.progress.recovery.operationId && detail.progress.recovery.operationId !== deployment.id}<a href={`/provisioning/${encodeURIComponent(detail.progress.recovery.operationId)}`}>View recovery operation →</a>{/if}</div>{/if}
{#if serverHref && !['done', 'cancelled'].includes(deployment.state)}<div class="diagnostics"><a href={`${serverHref}?tab=logs`}>Open server logs</a><a href={`${serverHref}?tab=diagnostics`}>Diagnose connection</a>{#if detail.progress?.phase === 'verify-map' || detail.progress?.checks.some(check => check.key === 'map' && check.result !== 'pass')}<a href={`${serverHref}?tab=config`}>Review map configuration</a>{/if}</div>{/if}
<div class="timing">
<div><span>{outcome.active ? 'Total elapsed' : 'Recorded duration'}</span><strong>{elapsedTime(deployment.started_at, deployment.finished_at ? Date.parse(deployment.finished_at) : now)}</strong></div>
{#if outcome.active}<div><span>Time in phase</span><strong>{elapsedTime(detail.progress?.phaseStartedAt, now)}</strong></div>{/if}
<div><span>Last evaluated</span><time datetime={detail.progress?.observedAt ?? undefined}>{fmt(detail.progress?.observedAt)}</time></div>
</div>
<div class="freshness"><span>Page updated <time datetime={view.refreshedAt ?? undefined}>{fmt(view.refreshedAt)}</time> · updates automatically</span><Button size="sm" variant="ghost" disabled={view.refreshing || !!view.busy} loading={view.refreshing} onclick={() => controller.forceRefresh()}>Refresh</Button></div>
</section>
{#key deployment.id}
<div class="detail-grid">
<div class="execution-column"><ProvisioningProgress {detail} /><ProvisioningActivity {detail} /></div>
<aside class="context-column" aria-label="Operation context">
<Card as="section" padding="md" ariaLabel="Operation details">
<div class="context-section"><h2>Operation details</h2><dl class="metadata">
<div><dt>Requested by</dt><dd>{deployment.triggered_by}</dd></div>
<div><dt>Requested at</dt><dd><time datetime={deployment.started_at}>{fmt(deployment.started_at)}</time></dd></div>
<div><dt>Submission attempted</dt><dd>{fmt(deployment.submitted_at)}</dd></div>
<div><dt>Finished</dt><dd>{deployment.finished_at ? fmt(deployment.finished_at) : outcome.active ? 'In progress' : 'Not recorded'}</dd></div>
{#if deployment.server_deleted_at}<div><dt>Identity deleted</dt><dd>{fmt(deployment.server_deleted_at)}</dd></div>{/if}
</dl></div>
</Card>
{#if detail.sources.length || detail.collection}<Card as="section" padding="md" ariaLabel="Related work"><div class="context-section"><h2>Related work</h2><p>The requesting work may have further checks after this operation finishes.</p><ul class="related">
{#if detail.collection}<li><a href={`/collections/launches/${encodeURIComponent(detail.collection.id)}`}>{detail.collection.name}</a><span>Collection launch</span></li>{/if}
{#each detail.sources as source (`${source.kind}:${source.source_id}`)}<li>{#if source.href}<a href={source.href}>{source.label}</a>{:else}<strong>{source.label}</strong><span>Source record no longer available</span>{/if}<code>{source.source_id}</code></li>{/each}
</ul></div></Card>{/if}
{#if detail.capacity.pools.length}<Card as="section" padding="md" ariaLabel="Capacity"><div class="context-section"><h2>Capacity</h2>
{#each detail.capacity.pools as pool (pool.kind)}<div class="capacity-pool"><div class="capacity-heading"><strong>{pool.kind === 'startup' ? 'Startup slots' : 'Restart recovery slots'}</strong><span>{pool.used} / {pool.limit} in use</span></div>
{#if pool.ownsSlot}<p>This operation holds a slot.</p>{/if}
{#if pool.requestOrder !== null}<p>Request order: {pool.requestOrder}. Eligible requests may pass blocked work; this is not a guaranteed start order.</p>{/if}
{#if pool.holders.length}<details><summary>Current slot holders</summary><ul>{#each pool.holders as holder}<li>{#if holder.id}<a href={`/provisioning/${encodeURIComponent(holder.id)}`}>{serverTitle(holder.server)}</a>{:else}<span>{serverTitle(holder.server)} · operation not recorded</span>{/if}</li>{/each}</ul></details>{/if}
</div>{/each}<p class="small">Capacity snapshot {fmt(detail.capacity.evaluatedAt)}</p>
</div></Card>{/if}
<Card as="section" padding="md" ariaLabel="Recorded launch inputs"><div class="context-section recipe"><h2>{recipeHeading}</h2>
{#if detail.recipe}
{#if detail.blueprintName}<p>From {detail.blueprintName}</p>{/if}
<p>{recipeStatus === 'queued' ? 'This capture does not pin versions. The latest promoted setup is selected again when creation begins.' : recipeStatus === 'attempted' ? 'These files were selected when creation began. Selection alone does not confirm successful installation.' : 'This is the saved recipe for this operation. The point at which it was selected was not recorded.'}</p>
<p class="small">Captured <time datetime={detail.recipe.capturedAt}>{fmt(detail.recipe.capturedAt)}</time></p>
<LaunchRecipe recipe={detail.recipe} historical selectionStatus={recipeStatus} />
<div class="launch-another"><Button variant="ghost" onclick={() => launchOpen = true}>Launch another server →</Button><p>Creates a separate server with a fresh ID and port using the latest promoted files. It does not repair this instance.</p></div>
{:else}<p>{['create', 'rebuild'].includes(deployment.action) ? 'No launch recipe was recorded for this operation.' : 'This operation uses the existing instance. No new launch files were selected.'}</p>{/if}
</div></Card>
</aside>
</div>
{#if detail.recipe}<DeployServerDialog bind:open={launchOpen} sourceDeploymentId={deployment.id} />{/if}
{/key}
{:else}<div class="loading" role="status">Loading provisioning operation…</div>{/if}
</div> </div>
<style> <style>
.recovery-links { display: flex; gap: 16px; flex-wrap: wrap; font-size: 14px; } .page { display: grid; gap: var(--space-6); min-width: 0; }
.page { display: grid; gap: 28px; } section { display: grid; gap: 16px; } .page :global(.subtitle), .page :global(.title) { overflow-wrap: anywhere; }
.summary { padding: 24px; border: 1px solid var(--border-subtle); border-radius: var(--radius-md); background: var(--surface); } .page :global(.btn) { white-space: normal; }
.saved-recipe { padding: 24px; border: 1px solid var(--border-subtle); border-radius: var(--radius-md); background: var(--surface-sunken); }.explanation { color: var(--text-secondary); max-width: 800px; }.timing { display: flex; gap: 24px; font-size: 12px; color: var(--text-secondary); }.timing strong { color: var(--text-primary); margin-left: 8px; font-family: var(--font-mono); font-weight: 500; } h2, h3, p { margin: 0; }a { color: var(--accent-primary-hover); text-underline-offset: 3px; }
.top { display: flex; align-items: center; justify-content: space-between; gap: 16px; flex-wrap: wrap; overflow-wrap: anywhere; } a:focus-visible, summary:focus-visible { outline: 2px solid var(--border-focus); outline-offset: 3px; border-radius: 2px; }
h2 { font-size: 18px; margin: 0; } p { margin: 0; line-height: 1.6; } a { color: var(--accent-primary-hover); } p { line-height: 1.65; overflow-wrap: anywhere; }
dl { display: flex; gap: 32px; flex-wrap: wrap; margin: 0; } dt, .muted { color: var(--text-secondary); font-size: 13px; } dd { margin: 6px 0 0; font-size: 14px; } .identity-row, .operation-id, .identity-links, .badges, .outcome-top, .diagnostics, .freshness, .recovery-actions { display: flex; align-items: center; gap: var(--space-3); flex-wrap: wrap; }
.identity-row { justify-content: space-between; margin-top: calc(-1 * var(--space-3)); gap: var(--space-2) var(--space-5); }.operation-id { gap: var(--space-2); min-width: 0; }.operation-id > span, .identity-links { font-size: var(--text-xs); color: var(--text-tertiary); }.identity-links { gap: var(--space-4); }
.error { color: var(--state-danger); } .warning { color: var(--state-warning); } .outcome { display: grid; gap: var(--space-5); padding: var(--space-6); border: 1px solid var(--border-strong); border-radius: var(--radius-lg); background: var(--surface); }
@media (max-width: 640px) { .outcome.attention { border-color: color-mix(in srgb, var(--state-warning) 35%, var(--border-strong)); }.outcome-top { justify-content: space-between; }.record-label { color: var(--text-tertiary); font-size: var(--text-xs); }.outcome h2 { font-size: var(--text-xl); line-height: 1.35; letter-spacing: var(--tracking-tight); overflow-wrap: anywhere; }
.timing { flex-direction: column; gap: var(--space-2); } .outcome-message { display: grid; gap: var(--space-2); }.outcome-message p { color: var(--text-secondary); font-size: var(--text-base); max-width: 85ch; }
} .failure, .action-error, .refresh-error { border: 1px solid color-mix(in srgb, var(--state-danger) 35%, var(--border-subtle)); border-radius: var(--radius-md); padding: var(--space-4); color: var(--state-danger); font-size: var(--text-sm); }.failure p, .action-error p { margin-top: var(--space-2); color: var(--text-primary); }.refresh-error { display: flex; justify-content: space-between; align-items: center; gap: var(--space-4); flex-wrap: wrap; }.refresh-error p { color: var(--text-secondary); margin-top: var(--space-1); }
.failure-excerpt { display: grid; gap: var(--space-1); font-size: var(--text-sm); color: var(--text-secondary); }.failure-excerpt span { font-size: var(--text-xs); color: var(--text-tertiary); }
.warnings { border-left: 3px solid var(--state-warning); padding-left: var(--space-4); font-size: var(--text-sm); }.warnings h3 { font-size: var(--text-sm); color: var(--state-warning); }.warnings ul { margin: var(--space-2) 0 0; padding-left: var(--space-4); line-height: 1.6; }.warnings li { overflow-wrap: anywhere; }
.notice { color: var(--state-online); font-size: var(--text-sm); }.action-help { color: var(--text-secondary); font-size: var(--text-xs); margin-top: var(--space-2); }.dependency, .diagnostics { font-size: var(--text-sm); }.diagnostics { gap: var(--space-3) var(--space-5); }
.recovery-note { display: grid; gap: var(--space-2); padding: var(--space-4); border: 1px solid var(--border-subtle); border-radius: var(--radius-md); background: var(--surface-sunken); font-size: var(--text-sm); }.recovery-note p, .recovery-note span { color: var(--text-secondary); }.recovery-note span { font-size: var(--text-xs); }
.timing { border-top: 1px solid var(--border-subtle); padding-top: var(--space-5); display: flex; flex-wrap: wrap; gap: var(--space-4) var(--space-8); }.timing > div { display: grid; gap: var(--space-2); }.timing span { font-size: var(--text-xs); color: var(--text-tertiary); }.timing strong { font-size: var(--text-md); font-variant-numeric: tabular-nums; font-weight: var(--weight-medium); }.timing time { font-size: var(--text-sm); color: var(--text-secondary); }
.freshness { justify-content: space-between; color: var(--text-tertiary); font-size: var(--text-xs); line-height: 1.65; margin-top: calc(-1 * var(--space-2)); }
.detail-grid { display: grid; grid-template-columns: minmax(0, 1fr) minmax(280px, 340px); gap: var(--space-8); align-items: start; }.execution-column { display: grid; gap: var(--space-8); min-width: 0; }.context-column { display: grid; gap: var(--space-5); min-width: 0; }.context-section { display: grid; gap: var(--space-4); min-width: 0; }.context-section h2 { font-size: var(--text-md); }.context-section p { font-size: var(--text-sm); color: var(--text-secondary); }.context-section p.small { font-size: var(--text-xs); }
.metadata { display: grid; gap: var(--space-4); margin: 0; }.metadata dt { color: var(--text-tertiary); font-size: var(--text-xs); }.metadata dd { margin: var(--space-1) 0 0; font-size: var(--text-sm); line-height: 1.6; overflow-wrap: anywhere; }
.related, .capacity-pool ul { list-style: none; padding: 0; margin: 0; display: grid; gap: var(--space-4); }.related li { display: grid; gap: var(--space-1); font-size: var(--text-sm); overflow-wrap: anywhere; }.related span, .related code { color: var(--text-tertiary); font-size: var(--text-xs); }.related strong { font-weight: var(--weight-medium); }code { font-family: var(--font-mono); }
.capacity-pool { display: grid; gap: var(--space-3); font-size: var(--text-xs); }.capacity-pool + .capacity-pool { border-top: 1px solid var(--border-subtle); padding-top: var(--space-4); }.capacity-heading { display: flex; justify-content: space-between; gap: var(--space-2); flex-wrap: wrap; }.capacity-heading strong { font-weight: var(--weight-medium); }.capacity-heading span { color: var(--text-secondary); }.capacity-pool summary { cursor: pointer; color: var(--text-secondary); }.capacity-pool ul { gap: var(--space-2); margin-top: var(--space-3); line-height: 1.6; }
.launch-another { display: grid; gap: var(--space-2); justify-items: start; border-top: 1px solid var(--border-subtle); padding-top: var(--space-4); }.launch-another p { font-size: var(--text-xs); }
.loading { min-height: 240px; padding: var(--space-6); border: 1px solid var(--border-subtle); border-radius: var(--radius-lg); color: var(--text-secondary); }
@media (max-width: 1000px) { .detail-grid { grid-template-columns: minmax(0, 1fr); }.context-column { grid-template-columns: repeat(2, minmax(0, 1fr)); } }
@media (max-width: 600px) { .page { gap: var(--space-5); }.outcome { padding: var(--space-4); gap: var(--space-4); }.outcome h2 { font-size: var(--text-lg); }.outcome-message p { font-size: var(--text-sm); }.context-column { grid-template-columns: minmax(0, 1fr); }.identity-row, .identity-links { align-items: flex-start; }.operation-id { display: block; }.operation-id > span { display: block; margin-bottom: var(--space-1); }.outcome-top { align-items: flex-start; }.record-label { display: none; }.timing { gap: var(--space-4); }.timing > div:last-child { flex-basis: 100%; }.freshness { align-items: flex-start; }.freshness > span { flex: 1 1 170px; }.detail-grid { gap: var(--space-6); } }
</style> </style>
+4 -1
View File
@@ -502,7 +502,10 @@ describe('server creation map selection', () => {
expect(komodoDestroyDeployment).not.toHaveBeenCalled(); expect(komodoDestroyDeployment).not.toHaveBeenCalled();
const detail = await (await GET({ params: { id: body.deploymentId } })).json(); const detail = await (await GET({ params: { id: body.deploymentId } })).json();
expect(detail.deployment).toMatchObject({ server: serverName, action: 'destroy', state: 'waiting' }); expect(detail.deployment).toMatchObject({ server: serverName, action: 'destroy', state: 'waiting' });
expect(detail.events).toHaveLength(2); expect(detail.events.map((event: { message: string }) => event.message)).toEqual(expect.arrayContaining([
'Server destruction requested', 'Waiting to verify the server is empty'
]));
expect(detail.events.every((event: { id: number }, index: number) => index === 0 || detail.events[index - 1].id < event.id)).toBe(true);
const missing = await DELETE({ params: { name: 'missing-server' } }); const missing = await DELETE({ params: { name: 'missing-server' } });
expect(missing.status).toBe(404); expect(missing.status).toBe(404);
}); });
+204
View File
@@ -0,0 +1,204 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { newFixture, type Fixture } from '../helpers/fixture.js';
let fixture: Fixture;
beforeEach(async () => { vi.resetModules(); fixture = await newFixture(); });
afterEach(async () => { await fixture.cleanup(); vi.restoreAllMocks(); });
describe('provisioning detail projection', () => {
it('records identical structured and legacy stage text once while preserving changed observations', async () => {
const { beginDeployment, deploymentStage, getDeployment } = await import('../../src/lib/server/provisioning/history.js');
const id = beginDeployment('server', 'restart');
const update = { phase: 'eligibility' as const,
blocker: { code: 'players', message: 'Waiting for 3 players', automatic: true } };
deploymentStage(id, update.blocker.message, update);
deploymentStage(id, update.blocker.message, update);
expect(getDeployment(id)!.events.filter(event => event.message === update.blocker.message)).toHaveLength(1);
deploymentStage(id, 'Waiting for 2 players', { ...update, blocker: { ...update.blocker, message: 'Waiting for 2 players' } });
const events = getDeployment(id)!.events;
expect(events.filter(event => event.message === 'Waiting for 2 players')).toHaveLength(1);
expect(events.filter(event => event.kind === 'blocker')).toHaveLength(1);
});
it('keeps legacy full activity while the page and opt-in API return a bounded snapshot without probing', async () => {
const history = await import('../../src/lib/server/provisioning/history.js');
const komodo = await import('../../src/lib/server/integrations/komodo.js');
const rcon = await import('../../src/lib/server/integrations/rcon.js');
const inspect = vi.spyOn(komodo, 'komodoInspectHealthy');
const status = vi.spyOn(rcon, 'rconStatusResult');
const id = history.beginDeployment('t5_zombie_theater_1', 'create');
for (let index = 0; index < 40; index++) history.recordDeploymentEvent(id, `Recorded activity ${index}`);
history.finishDeployment(id, { ok: false, message: 'A recorded startup failure' });
for (let index = 0; index < 8; index++) history.recordDeploymentEvent(id, `Later observation ${index}`);
const full = history.getDeployment(id)!;
const recent = history.getDeployment(id, { events: 'recent' })!;
expect(full.events).toHaveLength(50);
expect(recent.events).toEqual(full.events.slice(-5));
expect(recent.activity).toMatchObject({ total: 50, latestId: full.events.at(-1)!.id,
failureExcerpt: { message: 'Failed: A recorded startup failure', kind: 'outcome' } });
expect(recent.progress?.phase).toBe('requested');
expect(recent.progress?.checkpoints.map(point => point.key)).toEqual(['requested']);
const { GET } = await import('../../src/routes/api/provisioning/[id]/+server.js');
expect((await (await GET({ params: { id } })).json()).events).toHaveLength(50);
expect((await (await GET({ params: { id }, url: new URL('http://localhost/?events=recent') })).json()).events).toHaveLength(5);
const { load } = await import('../../src/routes/provisioning/[id]/+page.server.js');
expect(load({ params: { id } })?.detail.events).toHaveLength(5);
expect(inspect).not.toHaveBeenCalled();
expect(status).not.toHaveBeenCalled();
});
it('retains missing sources, normalizes malformed warnings, and leaves legacy evidence unrecorded', async () => {
const { db } = await import('../../src/lib/server/db/index.js');
const { getDeployment } = await import('../../src/lib/server/provisioning/history.js');
db().prepare(`INSERT INTO server_deployments (id,server,action,state,stage,started_at,triggered_by,warnings)
VALUES ('legacy','old-server','restart','done','Old custom stage','2026-01-01','ui',?)`).run('["Check files",7,null,{"bad":true}]');
db().prepare("INSERT INTO provisioning_sources VALUES ('legacy','campaign','missing','old-server')").run();
db().prepare("INSERT INTO provisioning_sources VALUES ('legacy','watchdog','incident','old-server')").run();
db().prepare(`INSERT INTO watchdog_incidents (id,server,kind,state,summary,first_seen_at,last_seen_at)
VALUES ('incident','old-server','health','resolved','Recovered game service','2026-01-01','2026-01-01')`).run();
const detail = getDeployment('legacy')!;
expect(detail.progress).toBeNull();
expect(detail.events).toEqual([]);
expect(detail.activity).toEqual({ total: 0, latestId: null, failureExcerpt: null });
expect(detail.warningMessages).toEqual(['Check files', 'Some recorded warnings could not be read.']);
expect(detail.sources).toEqual([
{ kind: 'campaign', source_id: 'missing', server: 'old-server', label: 'Campaign missing', href: null },
{ kind: 'watchdog', source_id: 'incident', server: 'old-server', label: 'Recovered game service', href: '/maintenance/incident' }
]);
db().prepare("UPDATE server_deployments SET warnings = 'broken json' WHERE id = 'legacy'").run();
expect(getDeployment('legacy')!.warningMessages).toEqual(['Recorded warnings could not be read.']);
});
it('correlates reservations by operation ID without assigning a later same-name slot to historical work', async () => {
const { db } = await import('../../src/lib/server/db/index.js');
const { beginDeployment, getDeployment, finishDeployment } = await import('../../src/lib/server/provisioning/history.js');
db().prepare("INSERT INTO servers (name,komodo_name) VALUES ('server','server')").run();
const old = beginDeployment('server', 'restart');
finishDeployment(old, { ok: false, message: 'Timeout' });
const current = beginDeployment('server', 'restart');
db().prepare("INSERT INTO restart_rollout (server,requested_at,operation_id) VALUES ('server','2026-01-01',?)").run(current);
expect(getDeployment(old)!.capacity.pools[0]).toMatchObject({ kind: 'restart', used: 1, ownsSlot: false,
holders: [{ id: current, server: 'server' }] });
expect(getDeployment(old)!.progress!.recovery).toBeNull();
expect(getDeployment(current)!.capacity.pools[0].ownsSlot).toBe(true);
db().prepare('UPDATE restart_rollout SET healthy_at = ?').run(new Date().toISOString());
expect(getDeployment(old)!.capacity.pools[0].used).toBe(0);
expect(getDeployment(old)!.deployment.state).toBe('failed');
});
it('labels submission order independently of startup reservations and retains queued recipe status', async () => {
const { db } = await import('../../src/lib/server/db/index.js');
const { beginDeployment, getDeployment } = await import('../../src/lib/server/provisioning/history.js');
const { recordProgress } = await import('../../src/lib/server/provisioning/progress.js');
const first = beginDeployment('first', 'create');
const next = beginDeployment('next', 'create');
db().prepare("UPDATE server_deployments SET state = 'queued' WHERE id = ?").run(next);
recordProgress(next, { phase: 'startup-queue', recipeStatus: 'queued', observedAt: null });
const detail = getDeployment(next)!;
expect(detail.capacity.pools[0]).toMatchObject({ kind: 'startup', used: 1, requestOrder: 1, ownsSlot: false,
holders: [{ id: first, server: 'first' }] });
expect(detail.progress).toMatchObject({ recipeStatus: 'queued', observedAt: null });
});
it('preserves a failure outcome and evidence before an observation-only same-ID resume', async () => {
const { db } = await import('../../src/lib/server/db/index.js');
const history = await import('../../src/lib/server/provisioning/history.js');
db().prepare("INSERT INTO servers (name,komodo_name,game_port) VALUES ('server','server',28960)").run();
const id = history.beginDeployment('server', 'create');
db().prepare("INSERT INTO server_creation_jobs VALUES (?, 'server',28960,'zombie','theater','zombie_theater')").run(id);
history.deploymentStage(id, 'Waiting for game evidence', { phase: 'verify-game', checks: [
{ key: 'game', label: 'Game status', result: 'waiting', source: null }
] });
history.finishDeployment(id, { ok: false, message: 'Observation interrupted' });
const failed = history.getDeployment(id)!;
expect(failed.progress?.phase).toBe('verify-game');
expect(history.resumeVerification(id)).toBe(true);
const resumed = history.getDeployment(id)!;
expect(resumed.deployment).toMatchObject({ id, state: 'awaiting-health', finished_at: null, error: null });
expect(resumed.progress).toMatchObject({ phase: 'verify-container', checks: [], observedAt: null });
const outcomes = resumed.events.filter(event => event.kind === 'outcome');
expect(outcomes).toHaveLength(1);
expect(JSON.parse(outcomes[0].details_json!)).toMatchObject({ state: 'failed', finishedAt: failed.deployment.finished_at,
error: 'Observation interrupted' });
expect(resumed.events.some(event => event.kind === 'resumed')).toBe(true);
});
it('preserves a legacy cleanup failure and keeps deleted cleanup capability authoritative', async () => {
const { db } = await import('../../src/lib/server/db/index.js');
const { getDeployment, resumeCleanup, REMOVING_SERVER } = await import('../../src/lib/server/provisioning/history.js');
db().prepare(`INSERT INTO server_deployments (id,server,server_deleted_at,action,state,stage,started_at,finished_at,triggered_by,error)
VALUES ('cleanup','old-server','2026-01-02','destroy','interrupted',?,'2026-01-01','2026-01-02','ui','App restarted')`).run(REMOVING_SERVER);
expect(getDeployment('cleanup')).toMatchObject({ canResumeCleanup: true, deployment: { server_exists: 0 } });
expect(resumeCleanup('cleanup')).toBe(true);
const result = getDeployment('cleanup')!;
expect(result.deployment).toMatchObject({ state: 'waiting', server_deleted_at: '2026-01-02' });
const outcome = result.events.find(event => event.kind === 'outcome')!;
expect(JSON.parse(outcome.details_json!)).toMatchObject({ state: 'interrupted', finishedAt: '2026-01-02', error: 'App restarted' });
expect(result.progress?.phase).toBe('cleanup');
});
});
describe('provisioning activity pagination', () => {
it('keeps polling payloads bounded for a retained history with thousands of events', async () => {
const { db } = await import('../../src/lib/server/db/index.js');
const { beginDeployment, getDeployment } = await import('../../src/lib/server/provisioning/history.js');
const id = beginDeployment('server', 'restart');
const insert = db().prepare('INSERT INTO server_deployment_events (deployment_id,created_at,message) VALUES (?,?,?)');
db().transaction(() => {
for (let index = 0; index < 5000; index++) insert.run(id, '2026-09-28T12:00:00Z', `Observation ${index} ${'x'.repeat(250)}`);
})();
const full = getDeployment(id)!;
const recent = getDeployment(id, { events: 'recent' })!;
const fullBytes = Buffer.byteLength(JSON.stringify(full));
const recentBytes = Buffer.byteLength(JSON.stringify(recent));
expect(recent.activity.total).toBe(5001);
expect(recent.events).toHaveLength(5);
expect(recentBytes).toBeLessThan(fullBytes / 100);
if (process.env.PROVISIONING_BENCHMARK) {
const start = performance.now();
for (let index = 0; index < 30; index++) getDeployment(id, { events: 'recent' });
console.info({ events: full.events.length, fullBytes, recentBytes, meanRecentReadMs: (performance.now() - start) / 30 });
}
});
it.each(['newest', 'oldest'] as const)('keeps stable %s pages under concurrent appends and searches literal text', async order => {
const { beginDeployment, recordDeploymentEvent } = await import('../../src/lib/server/provisioning/history.js');
const { deploymentActivity } = await import('../../src/lib/server/provisioning/activity.js');
const id = beginDeployment('server', 'stop');
const other = beginDeployment('other', 'stop');
for (let index = 0; index < 67; index++) {
recordDeploymentEvent(id, index === 30 ? 'literal %_needle' : `Event ${index}`);
recordDeploymentEvent(other, 'Unrelated event');
}
let page = deploymentActivity(id, { order });
const watermark = page.watermark;
const seen = [...page.events];
expect(page.events).toHaveLength(25);
recordDeploymentEvent(id, 'New event during reading');
while (page.nextCursor !== null) {
page = deploymentActivity(id, { order, watermark, cursor: page.nextCursor });
seen.push(...page.events);
}
expect(seen).toHaveLength(68);
expect(new Set(seen.map(event => event.id)).size).toBe(68);
expect(seen.every(event => event.id <= watermark)).toBe(true);
expect(page.total).toBe(68);
expect(page.totalEvents).toBe(69);
expect(deploymentActivity(id, { query: '%_NEEDLE' }).events.map(event => event.message)).toEqual(['literal %_needle']);
expect(deploymentActivity(id, { query: 'not found' }).total).toBe(0);
});
it('validates cursor/search requests and caps page size without changing default legacy responses', async () => {
const { beginDeployment, recordDeploymentEvent } = await import('../../src/lib/server/provisioning/history.js');
const { GET } = await import('../../src/routes/api/provisioning/[id]/events/+server.js');
const id = beginDeployment('server', 'stop');
for (let index = 0; index < 130; index++) recordDeploymentEvent(id, `Event ${index}`);
const response = GET({ params: { id }, url: new URL('http://localhost/?limit=999') });
expect((await response.json()).events).toHaveLength(100);
for (const query of ['cursor=-1', 'watermark=NaN', 'limit=0', 'order=invalid', 'cursor=9007199254740992']) {
expect(() => GET({ params: { id }, url: new URL(`http://localhost/?${query}`) })).toThrow();
}
expect(() => GET({ params: { id: 'missing' }, url: new URL('http://localhost/') })).toThrow();
});
});
+284
View File
@@ -0,0 +1,284 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { join } from 'node:path';
import { newFixture, type Fixture } from '../helpers/fixture.js';
import { gameEvent } from '../helpers/telemetry.js';
vi.mock('../../src/lib/server/integrations/komodo.js', () => ({
komodoRestart: vi.fn(), komodoRuntimeAction: vi.fn(), komodoInspectHealthy: vi.fn(),
komodoCreateDeployment: vi.fn(), komodoDeploy: vi.fn(), komodoDestroyDeployment: vi.fn(), komodoDeleteDeployment: vi.fn(),
komodoEnsureTag: vi.fn(), komodoUpdateTagsOnResource: vi.fn()
}));
vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconGetPlayerCount: vi.fn(), rconStatusResult: vi.fn() }));
vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}) }));
let fixture: Fixture;
const server = 't5_zombie_theater_1';
const status = (map = 'zombie_theater') => ({ ok: true as const, endpoint: { host: 'test', port: 28960 }, rawReply: '',
status: { players: 0, map, gametype: 'zom', roster: [] } });
beforeEach(async () => {
vi.resetModules(); vi.resetAllMocks();
vi.useFakeTimers({ toFake: ['Date'] });
vi.setSystemTime(new Date('2026-09-28T12:00:00Z'));
fixture = await newFixture([server]);
for (const [key, value] of Object.entries({ KOMODO_ADDRESS: 'http://komodo.test', KOMODO_API_KEY: 'test', KOMODO_API_SECRET: 'test',
KOMODO_TARGET_SERVER: 'host', RCON_PASSWORD: 'test', DEPLOYMENT_CREATE_CONCURRENCY: '1', DEPLOYMENT_RESTART_CONCURRENCY: '1',
SERVER_PORT_MIN: '28960', SERVER_PORT_MAX: '28970' })) vi.stubEnv(key, value);
const komodo = await import('../../src/lib/server/integrations/komodo.js');
for (const name of ['komodoRestart', 'komodoRuntimeAction', 'komodoCreateDeployment', 'komodoDeploy', 'komodoDestroyDeployment', 'komodoDeleteDeployment'] as const) {
vi.mocked(komodo[name]).mockResolvedValue({ ok: true });
}
vi.mocked(komodo.komodoInspectHealthy).mockResolvedValue({ ok: true, state: 'running' });
vi.mocked(komodo.komodoEnsureTag).mockImplementation(async name => ({ ok: true, id: name }));
vi.mocked(komodo.komodoUpdateTagsOnResource).mockResolvedValue({ ok: true });
const rcon = await import('../../src/lib/server/integrations/rcon.js');
vi.mocked(rcon.rconGetPlayerCount).mockResolvedValue(0);
vi.mocked(rcon.rconStatusResult).mockResolvedValue(status());
});
afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); vi.unstubAllEnvs(); });
async function register() {
const { db } = await import('../../src/lib/server/db/index.js');
db().prepare(`INSERT INTO servers (name,komodo_type,komodo_name,game,map,game_mode,map_slug,game_port,host_dir)
VALUES (?, 'deployment', ?, 't5','zombie_theater','zombie','theater',28960,?)`).run(server, server, join(fixture.t5, server));
const { saveServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
saveServerKey(server, 'saved-key');
}
async function detail(id: string) {
const { getDeployment } = await import('../../src/lib/server/provisioning/history.js');
return getDeployment(id)!;
}
async function telemetry(players: number, map = 'zombie_theater') {
const { db } = await import('../../src/lib/server/db/index.js');
const receivedAt = new Date().toISOString();
db().prepare(`INSERT INTO server_telemetry (server,round,players,received_at,payload) VALUES (?,1,?,?,?)
ON CONFLICT(server) DO UPDATE SET players=excluded.players,received_at=excluded.received_at,payload=excluded.payload`)
.run(server, players, receivedAt, JSON.stringify({ ...gameEvent('heartbeat', 1, players, server, map), receivedAt }));
return receivedAt;
}
describe('provisioning decision evidence', () => {
it('distinguishes queued recipes, attempted files, transient map checks, and telemetry completion without replaying startup', async () => {
const { queueCreateServer, processCreations } = await import('../../src/lib/server/provisioning/instances.js');
const { komodoCreateDeployment, komodoDeploy, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
const { rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js');
const job = queueCreateServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'saved-key' });
if (!job.ok) throw new Error(job.message);
expect((await detail(job.deploymentId)).progress).toMatchObject({ phase: 'startup-queue', recipeStatus: 'queued', observedAt: null });
const { createScript, getLatestVersion } = await import('../../src/lib/server/fleet/scripts.js');
const { db } = await import('../../src/lib/server/db/index.js');
const { script } = createScript({ name: 'New promotion', targetRelPath: 'promoted.gsc', scopeType: 'game', scopeValue: 't5', content: Buffer.from('promoted while queued') });
db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(getLatestVersion(script.id)!.id, script.id);
await processCreations();
expect((await detail(job.deploymentId)).progress?.recipeStatus).toBe('attempted');
expect((await detail(job.deploymentId)).recipe?.files.some(file => file.path === 'promoted.gsc')).toBe(true);
vi.mocked(rconStatusResult).mockResolvedValueOnce(status('zombie_moon'));
vi.setSystemTime(Date.now() + 1000);
await processCreations();
expect((await detail(job.deploymentId)).progress).toMatchObject({ phase: 'verify-map', blocker: { code: 'map-mismatch' },
checks: [{ key: 'container', result: 'pass' }, { key: 'game', result: 'pass' }, { key: 'map', result: 'waiting', expected: 'zombie_theater', actual: 'zombie_moon' }] });
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: false, reason: 'container recovering' });
await processCreations();
expect((await detail(job.deploymentId)).progress?.checks).toEqual([expect.objectContaining({ key: 'container', result: 'waiting' })]);
vi.setSystemTime(Date.now() + 1000);
const receivedAt = await telemetry(0);
await processCreations();
const completed = await detail(job.deploymentId);
expect(completed.deployment.state).toBe('done');
expect(completed.progress?.checks).toContainEqual(expect.objectContaining({ key: 'game', source: 'telemetry', result: 'pass', receivedAt }));
expect(completed.progress?.checkpoints).toContainEqual(expect.objectContaining({ key: 'verify-map' }));
expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
expect(komodoDeploy).toHaveBeenCalledTimes(1);
});
it('keeps an uncertain failed request immutable while observing and releasing its exact recovery slot', async () => {
await register();
const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js');
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoRestart).mockResolvedValueOnce({ ok: false, kind: 'unreachable', message: 'response lost' });
const job = requestLifecycle(server, 'restart');
if (!job.ok) throw new Error(job.error);
await processProvisioning();
const failed = await detail(job.deploymentId);
expect(failed.progress).toMatchObject({ command: { state: 'uncertain' }, recovery: { state: 'held', operationId: job.deploymentId } });
const failureEvents = failed.events.filter(event => event.kind === 'outcome');
vi.setSystemTime(Date.now() + 15_000);
await telemetry(0);
await processProvisioning();
const recovered = await detail(job.deploymentId);
expect(recovered.deployment).toMatchObject({ state: 'failed', finished_at: failed.deployment.finished_at });
expect(recovered.progress).toMatchObject({ command: { state: 'uncertain' }, recovery: { state: 'released', operationId: job.deploymentId } });
expect(recovered.events.filter(event => event.kind === 'outcome')).toEqual(failureEvents);
expect(komodoRestart).toHaveBeenCalledTimes(1);
});
it('records occupied and unknown evidence without treating either as zero or resetting eligibility timing', async () => {
await register();
const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js');
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
const job = requestLifecycle(server, 'restart');
if (!job.ok) throw new Error(job.error);
vi.mocked(rconGetPlayerCount).mockResolvedValueOnce(null);
await processProvisioning();
const unknown = await detail(job.deploymentId);
expect(unknown.progress?.checks).toEqual([expect.objectContaining({ key: 'occupancy', actual: null, source: null, result: 'unknown' })]);
vi.setSystemTime(Date.now() + 1000);
await telemetry(3);
await processProvisioning();
expect((await detail(job.deploymentId)).progress).toMatchObject({ phaseStartedAt: unknown.progress?.phaseStartedAt,
blocker: { code: 'players-present' }, checks: [{ key: 'occupancy', actual: 3, source: 'telemetry' }] });
vi.setSystemTime(Date.now() + 1000);
await telemetry(2);
await processProvisioning();
expect((await detail(job.deploymentId)).progress?.phaseStartedAt).toBe(unknown.progress?.phaseStartedAt);
expect(komodoRestart).not.toHaveBeenCalled();
});
it('preserves failed cleanup evidence through same-ID resume without another destructive command', async () => {
await register();
const { db } = await import('../../src/lib/server/db/index.js');
const { destroyServer, processDestructions } = await import('../../src/lib/server/provisioning/instances.js');
const { resumeCleanup } = await import('../../src/lib/server/provisioning/history.js');
const { komodoInspectHealthy, komodoDestroyDeployment, komodoDeleteDeployment } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'exited', reason: 'stopped' });
db().exec("CREATE TRIGGER fail_cleanup BEFORE DELETE ON servers BEGIN SELECT RAISE(ABORT, 'cleanup interrupted'); END");
const job = destroyServer(server);
if (!job.ok || !job.deploymentId) throw new Error('Could not queue removal');
await processDestructions();
const failed = await detail(job.deploymentId);
expect(failed.deployment.state).toBe('failed');
expect(failed.progress?.checkpoints).toContainEqual(expect.objectContaining({ key: 'delete-deployment' }));
expect(failed.progress?.checkpoints.some(point => point.key === 'cleanup')).toBe(false);
const failure = failed.events.find(event => event.kind === 'outcome')!;
db().exec('DROP TRIGGER fail_cleanup');
vi.setSystemTime(Date.now() + 1000);
expect(resumeCleanup(job.deploymentId)).toBe(true);
await processDestructions();
const completed = await detail(job.deploymentId);
expect(completed.deployment).toMatchObject({ id: job.deploymentId, state: 'done', server_exists: 0 });
expect(completed.events).toContainEqual(failure);
expect(completed.events.some(event => event.kind === 'resumed')).toBe(true);
expect(completed.progress?.checkpoints).toContainEqual(expect.objectContaining({ key: 'cleanup' }));
expect(komodoDestroyDeployment).toHaveBeenCalledTimes(1);
expect(komodoDeleteDeployment).toHaveBeenCalledTimes(1);
});
it('links the actual removal dependency while later eligible creation bypasses it', async () => {
const { db } = await import('../../src/lib/server/db/index.js');
const { beginDeployment } = await import('../../src/lib/server/provisioning/history.js');
const { queueCreateServer, processCreations } = await import('../../src/lib/server/provisioning/instances.js');
const input = { gameMode: 'zombie' as const, mapSlug: 'theater', plutoKey: 'saved-key' };
const first = queueCreateServer(input), second = queueCreateServer(input);
if (!first.ok || !second.ok) throw new Error('Could not queue creation');
const removal = beginDeployment('old-instance', 'destroy');
db().prepare('INSERT INTO server_creation_dependencies VALUES (?, ?)').run(first.deploymentId, removal);
await processCreations();
expect((await detail(first.deploymentId)).progress?.blocker).toMatchObject({ code: 'removal-dependency', href: `/provisioning/${removal}` });
expect((await detail(first.deploymentId)).deployment.state).toBe('queued');
expect((await detail(second.deploymentId)).deployment.state).toBe('awaiting-health');
});
it.each(['create', 'delete'] as const)('records uncertainty when the %s reply throws without inventing a successful checkpoint', async command => {
const komodo = await import('../../src/lib/server/integrations/komodo.js');
const { queueCreateServer, processCreations, destroyServer, processDestructions } = await import('../../src/lib/server/provisioning/instances.js');
let id: string;
if (command === 'create') {
vi.mocked(komodo.komodoCreateDeployment).mockRejectedValueOnce(new Error('reply lost'));
const job = queueCreateServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'saved-key' });
if (!job.ok) throw new Error(job.message);
id = job.deploymentId;
await processCreations();
expect(komodo.komodoCreateDeployment).toHaveBeenCalledTimes(1);
} else {
await register();
vi.mocked(komodo.komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'exited', reason: 'stopped' });
vi.mocked(komodo.komodoDeleteDeployment).mockRejectedValueOnce(new Error('reply lost'));
const job = destroyServer(server);
if (!job.ok || !job.deploymentId) throw new Error('Could not queue removal');
id = job.deploymentId;
await processDestructions();
expect(komodo.komodoDeleteDeployment).toHaveBeenCalledTimes(1);
}
const failed = await detail(id);
expect(failed.deployment.state).toBe('failed');
expect(failed.progress?.command).toMatchObject({ state: 'uncertain', message: expect.stringContaining('reply lost') });
expect(failed.progress?.checkpoints.some(point => point.key === (command === 'create' ? 'create-deployment' : 'delete-deployment'))).toBe(false);
});
it.each(['destroy', 'rebuild'] as const)('retains the resumable %s checkpoint when a beta arrives after confirmed remote deletion', async action => {
await register();
const { db } = await import('../../src/lib/server/db/index.js');
const { createScript } = await import('../../src/lib/server/fleet/scripts.js');
const { destroyServer, processDestructions } = await import('../../src/lib/server/provisioning/instances.js');
const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js');
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
const { resumeCleanup, resumeRebuild, REMOVING_SERVER } = await import('../../src/lib/server/provisioning/history.js');
const { komodoInspectHealthy, komodoDestroyDeployment, komodoDeleteDeployment } = await import('../../src/lib/server/integrations/komodo.js');
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'exited', reason: 'stopped' });
vi.mocked(komodoDeleteDeployment).mockImplementationOnce(async () => {
const { script, version } = createScript({ name: 'Late beta', targetRelPath: 'beta.gsc', scopeType: 'game', scopeValue: 't5', content: Buffer.from('beta') });
const beta = db().prepare("INSERT INTO betas (script_id,script_version_id,trigger_kind,state) VALUES (?,?,'manual','running')").run(script.id, version.id).lastInsertRowid;
db().prepare("INSERT INTO beta_servers (beta_id,server,had_original,stage) VALUES (?,?,0,'ready')").run(beta, server);
return { ok: true };
});
const job = action === 'destroy' ? destroyServer(server) : requestLifecycle(server, 'rebuild');
if (!job.ok || !job.deploymentId) throw new Error('Could not queue removal');
await (action === 'destroy' ? processDestructions() : processProvisioning());
const failed = await detail(job.deploymentId);
expect(failed.deployment).toMatchObject({ state: 'failed', stage: REMOVING_SERVER });
expect(failed.progress?.checkpoints).toContainEqual(expect.objectContaining({ key: 'delete-deployment' }));
expect(action === 'destroy' ? failed.canResumeCleanup : failed.canResumeRebuild).toBe(true);
db().prepare("UPDATE betas SET state = 'rolled-back'").run();
expect(action === 'destroy' ? resumeCleanup(job.deploymentId) : resumeRebuild(job.deploymentId)).toBe(true);
await (action === 'destroy' ? processDestructions() : processProvisioning());
expect((await detail(job.deploymentId)).deployment.state).toBe(action === 'destroy' ? 'done' : 'queued');
expect(komodoDestroyDeployment).toHaveBeenCalledTimes(1);
expect(komodoDeleteDeployment).toHaveBeenCalledTimes(1);
});
it('ignores occupancy evidence that returns after the request was cancelled', async () => {
await register();
const { requestLifecycle, cancelProvisioning } = await import('../../src/lib/server/provisioning/requests.js');
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js');
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
const job = requestLifecycle(server, 'restart');
if (!job.ok) throw new Error(job.error);
vi.mocked(rconGetPlayerCount).mockImplementationOnce(async () => {
expect(cancelProvisioning(job.deploymentId)).toBe(true);
return 4;
});
await processProvisioning();
const cancelled = await detail(job.deploymentId);
expect(cancelled.deployment.state).toBe('cancelled');
expect(cancelled.progress).toMatchObject({ checks: [], blocker: null });
expect(komodoRestart).not.toHaveBeenCalled();
});
it('does not attach replacement health to a deleted creation after a delayed probe', async () => {
const { db } = await import('../../src/lib/server/db/index.js');
const { queueCreateServer, processCreations } = await import('../../src/lib/server/provisioning/instances.js');
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
const { rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js');
const job = queueCreateServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'saved-key' });
if (!job.ok) throw new Error(job.message);
await processCreations();
vi.mocked(komodoInspectHealthy).mockImplementationOnce(async () => {
db().prepare('DELETE FROM servers WHERE name = ?').run(server);
await register();
return { ok: true, state: 'running' };
});
await processCreations();
const old = await detail(job.deploymentId);
expect(old.deployment.server_deleted_at).toBeTruthy();
expect(old.progress?.checks).toEqual([]);
expect(old.progress?.checkpoints.some(point => point.key.startsWith('verify-'))).toBe(false);
expect(rconStatusResult).not.toHaveBeenCalled();
});
});
@@ -0,0 +1,131 @@
import { describe, expect, it } from 'vitest';
import type { ProvisioningDetail, ProvisioningProgress } from '../../src/lib/provisioning-detail.js';
import { provisioningActions } from '../../src/lib/provisioning.js';
import { isProvisioningDetail, isProvisioningEventPage, provisioningGroups, provisioningOutcome, provisioningReturn } from '../../src/lib/ui/provisioning-presentation.js';
const at = '2026-09-28T12:00:00Z';
function progress(patch: Partial<ProvisioningProgress> = {}): ProvisioningProgress {
return { version: 1, phase: 'verify-map', phaseStartedAt: at, observedAt: at, blocker: null, checks: [], checkpoints: [],
command: null, recovery: null, recipeStatus: null, ...patch };
}
function fixture(): ProvisioningDetail {
return {
deployment: { id: 'operation-a', server: 't5_zombie_theater_1', action: 'create', state: 'awaiting-health', stage: 'Waiting for a map',
server_deleted_at: null, server_exists: 1, started_at: at, finished_at: null, triggered_by: 'ui', error: null, warnings: '[]' },
generatedAt: at, events: [], activity: { total: 0, latestId: null, failureExcerpt: null }, sources: [],
recipe: null, blueprintName: null, canResumeCleanup: false, canResumeRebuild: false, canResumeVerification: false, canCancelRemoval: false,
collection: null, progress: progress(), capacity: { evaluatedAt: at, pools: [] }, warningMessages: []
};
}
describe('provisioning outcome and evidence presentation', () => {
it.each(provisioningActions)('describes cancelled %s as terminal, with no automatic execution claim', action => {
const detail = fixture();
detail.deployment.action = action;
detail.deployment.state = 'cancelled';
const outcome = provisioningOutcome(detail);
expect(outcome.active).toBe(false);
expect(outcome.headline).toContain('cancelled');
expect(outcome.description).toContain('remaining work will not run');
expect(outcome.description).not.toMatch(/verifying|automatically/);
});
it('keeps backend-authorized cleanup actionable and active after identity deletion', () => {
const detail = fixture();
detail.deployment.action = 'destroy';
detail.deployment.state = 'failed';
detail.deployment.server_deleted_at = at;
detail.deployment.server_exists = 0;
detail.canResumeCleanup = true;
expect(provisioningOutcome(detail).description).toContain('Resume the remaining local cleanup');
detail.deployment.state = 'waiting';
detail.canResumeCleanup = false;
detail.progress = progress({ phase: 'cleanup', checks: [] });
expect(provisioningOutcome(detail).active).toBe(true);
expect(provisioningOutcome(detail).description).toContain('without repeating container removal');
expect(provisioningOutcome(detail).description).not.toContain('eligibility');
});
it('does not invent completed phases from a later phase or final outcome', () => {
const detail = fixture();
const steps = provisioningGroups(detail).flatMap(group => group.steps);
expect(steps.find(step => step.key === 'prepare')?.state).toBe('unknown');
expect(steps.find(step => step.key === 'verify-map')?.state).toBe('current');
detail.deployment.state = 'done';
expect(provisioningGroups(detail).flatMap(group => group.steps).every(step => step.state === 'unknown')).toBe(true);
detail.progress = null;
expect(provisioningGroups(detail)).toEqual([]);
expect(provisioningOutcome(detail).description).toContain('evidence was not recorded');
});
it('shows passed container and telemetry game checks while waiting on the engine map', () => {
const detail = fixture();
detail.progress = progress({ checks: [
{ key: 'container', label: 'Container readiness', result: 'pass', source: 'komodo', checkedAt: at, receivedAt: null, expected: 'healthy', actual: 'healthy', message: null },
{ key: 'game', label: 'Game status', result: 'pass', source: 'telemetry', checkedAt: at, receivedAt: at, expected: null, actual: null, message: 'Fresh telemetry accepted' },
{ key: 'map', label: 'Engine map', result: 'waiting', source: 'telemetry', checkedAt: at, receivedAt: at, expected: 'zombie_theater', actual: 'zombie_moon', message: null }
] });
const steps = provisioningGroups(detail).flatMap(group => group.steps);
expect(steps.find(step => step.key === 'verify-container')?.state).toBe('done');
expect(steps.find(step => step.key === 'verify-game')?.state).toBe('done');
expect(steps.find(step => step.key === 'verify-map')?.state).toBe('waiting');
expect(steps.find(step => step.key === 'verify-game')?.label).not.toContain('RCON');
detail.deployment.state = 'done';
expect(provisioningOutcome(detail).description).not.toContain('RCON');
});
it('newer unknown readiness evidence supersedes an old completion checkpoint', () => {
const detail = fixture();
detail.progress = progress({ checkpoints: [{ key: 'verify-game', label: 'Game passed', at, phase: 'verify-game' }],
checks: [{ key: 'game', label: 'Game status', result: 'unknown', source: null, checkedAt: '2026-09-28T12:01:00Z', receivedAt: null, expected: null, actual: null, message: 'No fresh evidence' }] });
expect(provisioningGroups(detail).flatMap(group => group.steps).find(step => step.key === 'verify-game')).toMatchObject({ state: 'unknown', at: '2026-09-28T12:01:00Z' });
});
it('preserves the failed request when later recovery checks pass', () => {
const detail = fixture();
detail.deployment.action = 'restart'; detail.deployment.state = 'failed';
detail.progress = progress({ phase: 'verify-game', recovery: { state: 'released', at, message: 'Recovery verified and capacity released.' },
checks: [{ key: 'game', label: 'Game status', result: 'pass', source: 'rcon', checkedAt: at, receivedAt: null, expected: null, actual: 0, message: null }] });
expect(provisioningOutcome(detail).headline).toBe('Restart failed');
expect(provisioningGroups(detail).flatMap(group => group.steps).find(step => step.key === 'verify-game')?.state).toBe('done');
});
it('keeps removal checkpoints separate from replacement creation', () => {
const detail = fixture(); detail.deployment.action = 'rebuild'; detail.deployment.state = 'waiting';
detail.progress = progress({ phase: 'cleanup', checkpoints: [{ key: 'delete-deployment', label: 'Remote deployment deleted', at, phase: 'delete-deployment' }] });
const groups = provisioningGroups(detail);
expect(groups.map(group => group.key)).toEqual(['request', 'remove', 'create', 'verify']);
expect(groups.flatMap(group => group.steps).find(step => step.key === 'delete-deployment')?.state).toBe('done');
expect(groups.flatMap(group => group.steps).find(step => step.key === 'cleanup')?.state).toBe('waiting');
expect(groups.flatMap(group => group.steps).find(step => step.key === 'prepare')?.state).toBe('pending');
});
});
describe('provisioning response and return-context boundaries', () => {
it('retains list filters without accepting an external or unrelated return destination', () => {
expect(provisioningReturn('/provisioning?view=history&action=restart&server=Lobby+1&limit=50&redirect=https://evil.test').href)
.toBe('/provisioning?view=history&action=restart&server=Lobby+1&limit=50');
for (const value of ['https://evil.test', '//evil.test', '/\\evil.test', '/\n/evil.test', '/settings', '/provisioning/other']) {
expect(provisioningReturn(value).href).toBe('/provisioning');
}
expect(provisioningReturn('/collections/launches/launch-a').href).toBe('/collections/launches/launch-a');
});
it('rejects incomplete nested snapshots before they replace a working detail', () => {
const detail = fixture();
expect(isProvisioningDetail(detail, detail.deployment.id)).toBe(true);
expect(isProvisioningDetail(detail, 'another-id')).toBe(false);
for (const patch of [
{ progress: undefined }, { progress: {} }, { progress: { ...detail.progress, checks: [null] } },
{ capacity: { evaluatedAt: at, pools: [{ kind: 'restart', holders: null }] } }, { activity: { total: 0 } },
{ warningMessages: {} }, { recipe: { files: [] } }, { sources: [{ kind: 'campaign', label: 'Campaign', source_id: 'a', href: 'javascript:alert(1)' }] }
]) expect(isProvisioningDetail({ ...detail, ...patch }, detail.deployment.id)).toBe(false);
expect(isProvisioningDetail({ ...detail, progress: { ...detail.progress, command: { state: 'rejected', at, message: 'Rejected before submission' } } }, detail.deployment.id)).toBe(true);
});
it('rejects an invalid history event without dropping the existing history view', () => {
const page = { events: [{ id: 1, created_at: at, message: 'Queued' }], total: 1, totalEvents: 1, latestId: 1, watermark: 1, nextCursor: null };
expect(isProvisioningEventPage(page)).toBe(true);
expect(isProvisioningEventPage({ ...page, events: [{ id: 1, created_at: 'invalid', message: 'Queued' }] })).toBe(false);
});
});
+111
View File
@@ -0,0 +1,111 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import Database from 'better-sqlite3';
import { readFile } from 'node:fs/promises';
import { newFixture, type Fixture } from '../helpers/fixture.js';
let fixture: Fixture;
beforeEach(async () => {
vi.resetModules();
fixture = await newFixture([]);
vi.useFakeTimers({ toFake: ['Date'] });
vi.setSystemTime(new Date('2026-09-28T12:00:00Z'));
});
afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); });
async function setup() {
const { db } = await import('../../src/lib/server/db/index.js');
const handle = db();
handle.prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by)
VALUES ('operation', 'server', 'create', 'waiting', 'Recorded stage', '2026-09-28T12:00:00Z', 'test')`).run();
const { recordProgress, readProgress } = await import('../../src/lib/server/provisioning/progress.js');
const read = () => readProgress(handle.prepare('SELECT observation_json FROM server_deployments WHERE id = ?').get('operation') as { observation_json: string });
return { handle, recordProgress, read };
}
describe('provisioning evidence persistence', () => {
it('keeps phase duration stable and avoids semantic event spam during repeated occupancy checks', async () => {
const { handle, recordProgress, read } = await setup();
for (let players = 4; players > 0; players--) {
recordProgress('operation', { phase: 'eligibility', blocker: { code: 'occupied', message: `${players} players remain`, automatic: true },
checks: [{ key: 'occupancy', label: 'Players', source: 'telemetry', result: 'waiting', actual: players,
receivedAt: new Date().toISOString() }] });
vi.setSystemTime(Date.now() + 15_000);
}
expect(read()).toMatchObject({ phaseStartedAt: '2026-09-28T12:00:00.000Z', observedAt: '2026-09-28T12:00:45.000Z',
checks: [expect.objectContaining({ actual: 1, receivedAt: '2026-09-28T12:00:45.000Z' })] });
expect(handle.prepare("SELECT count(*) AS count FROM server_deployment_events WHERE kind = 'blocker'").get()).toEqual({ count: 1 });
recordProgress('operation', { phase: 'container-action', blocker: null,
checkpoint: { key: 'eligibility', label: 'Eligibility checked', phase: 'eligibility' } });
expect(read()).toMatchObject({ phaseStartedAt: '2026-09-28T12:01:00.000Z', blocker: null });
});
it('replaces transient checks while retaining confirmed milestones and prior command uncertainty', async () => {
const { handle, recordProgress, read } = await setup();
recordProgress('operation', { phase: 'verify-game', command: { state: 'uncertain', message: 'Reply was lost' },
checkpoint: { key: 'container-action', label: 'Submission attempted', phase: 'container-action' },
checks: [{ key: 'container', label: 'Container', source: 'komodo', result: 'pass' },
{ key: 'game', label: 'Game', source: 'telemetry', result: 'pass' }] });
vi.setSystemTime(Date.now() + 15_000);
recordProgress('operation', { phase: 'verify-container', checks: [{ key: 'container', label: 'Container', source: 'komodo', result: 'unknown' }] });
expect(read()?.checks).toHaveLength(1);
expect(read()?.checks[0].result).toBe('unknown');
expect(read()?.checkpoints).toHaveLength(1);
expect(read()?.command?.state).toBe('uncertain');
expect(handle.prepare("SELECT message FROM server_deployment_events WHERE kind = 'command'").get()).toEqual({ message: 'Reply was lost' });
});
it('keeps operation-owned recovery evidence after slot/server removal and does not rewrite the failed outcome', async () => {
const { handle, recordProgress, read } = await setup();
handle.prepare("UPDATE server_deployments SET state = 'failed', finished_at = '2026-09-28T12:00:00Z' WHERE id = 'operation'").run();
recordProgress('operation', { recovery: { state: 'held', message: 'Observing possible submission' } });
vi.setSystemTime(Date.now() + 30_000);
recordProgress('operation', { recovery: { state: 'released', message: 'Recovery verified' } });
recordProgress('operation', { recovery: { state: 'released', message: 'Recovery verified' } });
expect(read()?.recovery).toMatchObject({ state: 'released', at: '2026-09-28T12:00:30.000Z' });
expect(handle.prepare("SELECT state, finished_at FROM server_deployments WHERE id = 'operation'").get())
.toEqual({ state: 'failed', finished_at: '2026-09-28T12:00:00Z' });
expect(handle.prepare("SELECT count(*) AS count FROM server_deployment_events WHERE kind = 'recovery'").get()).toEqual({ count: 2 });
});
it('rolls back state, snapshot, and milestone events together with its caller transaction', async () => {
const { handle, recordProgress, read } = await setup();
expect(() => handle.transaction(() => {
handle.prepare("UPDATE server_deployments SET state = 'running' WHERE id = 'operation'").run();
recordProgress('operation', { phase: 'cleanup', checkpoint: { key: 'delete-deployment', label: 'Remote deletion confirmed' } });
throw new Error('Interrupted transaction');
})()).toThrow('Interrupted transaction');
expect(read()).toBeNull();
expect(handle.prepare("SELECT state FROM server_deployments WHERE id = 'operation'").get()).toEqual({ state: 'waiting' });
expect(handle.prepare('SELECT * FROM server_deployment_events').all()).toEqual([]);
});
it('treats unsupported or malformed historical snapshots as unrecorded', async () => {
const { readProgress } = await import('../../src/lib/server/provisioning/progress.js');
for (const observation_json of [null, '', '[]', '{broken', '{"version":2}', '{"version":1,"checks":"pass"}']) {
expect(readProgress({ observation_json })).toBeNull();
}
});
it('adds nullable evidence columns without losing historical identity, row order, or events across reopen', async () => {
const schema = (await readFile(new URL('../../src/lib/server/db/schema.sql', import.meta.url), 'utf8'))
.replace(',\n phase_key TEXT,\n phase_started_at TEXT,\n observed_at TEXT,\n observation_json TEXT', '')
.replace(',\n kind TEXT,\n phase_key TEXT,\n details_json TEXT', '')
.replace("'create', 'start', 'restart', 'rebuild', 'stop', 'destroy'", "'create', 'destroy'");
const old = new Database(fixture.dbPath);
old.exec(schema);
old.prepare(`INSERT INTO server_deployments (rowid,id,server,server_deleted_at,action,state,stage,started_at,triggered_by)
VALUES (42,'legacy','server','2026-09-01','create','done','Recorded result','2026-08-01','test')`).run();
old.prepare("INSERT INTO server_deployment_events (id,deployment_id,created_at,message) VALUES (19,'legacy','2026-08-01','Original evidence')").run();
old.close();
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
expect(db().prepare("SELECT rowid, server_deleted_at, observation_json FROM server_deployments WHERE id = 'legacy'").get())
.toEqual({ rowid: 42, server_deleted_at: '2026-09-01', observation_json: null });
const { recordProgress, readProgress } = await import('../../src/lib/server/provisioning/progress.js');
recordProgress('legacy', { recovery: { state: 'unknown', message: 'Historical slot release was not recorded' } });
closeDb();
const saved = db().prepare("SELECT observation_json FROM server_deployments WHERE id = 'legacy'").get() as { observation_json: string };
expect(readProgress(saved)?.recovery?.state).toBe('unknown');
expect(db().prepare('SELECT id, message FROM server_deployment_events WHERE id = 19').get()).toEqual({ id: 19, message: 'Original evidence' });
expect(db().pragma('foreign_key_check')).toEqual([]);
});
});
+119
View File
@@ -0,0 +1,119 @@
import { afterEach, describe, expect, it, vi } from 'vitest';
import { createProvisioningRefresh, type ProvisioningRefreshState } from '../../src/lib/ui/provisioning-refresh.js';
type Detail = { id: string; revision: number };
const detail = (id: string, revision = 1): Detail => ({ id, revision });
const response = (id: string, revision = 1) => Response.json(detail(id, revision));
const controllers: { dispose: () => void }[] = [];
afterEach(() => { controllers.splice(0).forEach(controller => controller.dispose()); vi.useRealTimers(); });
function setup(fetcher: typeof fetch) {
let state!: ProvisioningRefreshState<Detail>;
const onChange = vi.fn(value => { state = value; });
const controller = createProvisioningRefresh<Detail>({ fetcher, onChange, timeoutMs: 100,
validate: (value, id): value is Detail => !!value && typeof value === 'object' && (value as Detail).id === id,
now: () => '2026-09-28T12:00:00Z' });
controllers.push(controller);
return { ...controller, onChange, state: () => state };
}
function deferred() {
let resolve!: (value: Response) => void;
let reject!: (error: Error) => void;
const promise = new Promise<Response>((done, fail) => { resolve = done; reject = fail; });
return { promise, resolve, reject };
}
describe('provisioning detail refresh ownership', () => {
it('uses the server snapshot without another initial request and coalesces background reads', async () => {
const pending = deferred();
const fetcher = vi.fn(() => pending.promise);
const page = setup(fetcher);
page.setOperation('a', detail('a'), '2026-09-28T11:59:00Z');
expect(fetcher).not.toHaveBeenCalled();
const first = page.refresh();
const second = page.refresh();
expect(fetcher).toHaveBeenCalledTimes(1);
pending.resolve(response('a', 2));
await Promise.all([first, second]);
expect(page.state()).toMatchObject({ detail: detail('a', 2), refreshing: false, refreshedAt: '2026-09-28T12:00:00Z' });
});
it('ignores obsolete success and failure across A → B → A, even when transport ignores abort', async () => {
const oldA = deferred(), oldB = deferred(), newA = deferred();
const fetcher = vi.fn().mockReturnValueOnce(oldA.promise).mockReturnValueOnce(oldB.promise).mockReturnValueOnce(newA.promise);
const page = setup(fetcher);
page.setOperation('a', detail('a'));
const first = page.refresh();
page.setOperation('b', detail('b'));
const second = page.refresh();
page.setOperation('a', detail('a', 3));
const third = page.refresh();
newA.resolve(response('a', 4));
await third;
oldA.resolve(response('a', 0));
oldB.reject(new Error('Obsolete failure'));
await Promise.all([first, second]);
expect(page.state()).toMatchObject({ id: 'a', detail: detail('a', 4), loadError: '', refreshing: false });
});
it('invalidates a pending poll and reconciles an accepted mutation before old data can return', async () => {
const old = deferred();
const fetcher = vi.fn().mockReturnValueOnce(old.promise).mockResolvedValueOnce(Response.json({ ok: true }))
.mockResolvedValueOnce(response('a', 3));
const page = setup(fetcher);
page.setOperation('a', detail('a'));
const polling = page.refresh();
await page.act('resume', 'Verification resumed.');
old.resolve(response('a', 2));
await polling;
expect(fetcher).toHaveBeenCalledTimes(3);
expect(page.state()).toMatchObject({ detail: detail('a', 3), notice: 'Verification resumed.', busy: null, loadError: '', actionError: '' });
});
it('retains accepted-action feedback if reconciliation fails, and action errors survive successful polls', async () => {
const fetcher = vi.fn().mockResolvedValueOnce(Response.json({ ok: true }))
.mockRejectedValueOnce(new Error('Offline'))
.mockResolvedValueOnce(Response.json({ error: 'Removal already began.' }, { status: 409 }))
.mockResolvedValueOnce(response('a', 2)).mockResolvedValueOnce(response('a', 3));
const page = setup(fetcher);
page.setOperation('a', detail('a'));
await page.act('resume', 'Resume accepted.');
expect(page.state()).toMatchObject({ notice: 'Resume accepted.', loadError: 'Offline', actionError: '', detail: detail('a') });
await page.act('cancel', 'Cancelled.');
await page.refresh();
expect(page.state()).toMatchObject({ actionError: 'Removal already began.', notice: '', loadError: '', detail: detail('a', 3) });
});
it('does not duplicate mutations or poll during them and ignores completion after navigation', async () => {
const action = deferred();
const fetcher = vi.fn(() => action.promise);
const page = setup(fetcher);
page.setOperation('a', detail('a'));
const first = page.act('resume', 'Resumed.');
await page.act('resume', 'Resumed twice.');
await page.refresh();
expect(fetcher).toHaveBeenCalledTimes(1);
page.setOperation('b', detail('b'));
action.resolve(Response.json({ ok: true }));
await first;
expect(fetcher).toHaveBeenCalledTimes(1);
expect(page.state()).toMatchObject({ id: 'b', busy: null, notice: '', actionError: '' });
});
it('times out reads without discarding the last snapshot and suppresses updates after unmount', async () => {
vi.useFakeTimers();
const fetcher = vi.fn((_url, init) => new Promise<Response>((_, reject) => {
init?.signal?.addEventListener('abort', () => reject(new Error('aborted')));
})) as typeof fetch;
const page = setup(fetcher);
page.setOperation('a', detail('a'));
const pending = page.refresh();
await vi.advanceTimersByTimeAsync(100);
await pending;
expect(page.state()).toMatchObject({ detail: detail('a'), loadError: 'Operation refresh timed out. Try again.', refreshing: false });
const next = page.refresh();
page.dispose();
const calls = page.onChange.mock.calls.length;
await next;
expect(page.onChange).toHaveBeenCalledTimes(calls);
});
});
+2 -1
View File
@@ -303,7 +303,8 @@ describe('Watchdog provisioning and telemetry evidence', () => {
expect(result.ok).toBe(true); expect(result.ok).toBe(true);
if (!result.ok) throw new Error(result.error); if (!result.ok) throw new Error(result.error);
expect(requestWatchdogLifecycle(server, 'restart', 'incident')).toEqual(result); expect(requestWatchdogLifecycle(server, 'restart', 'incident')).toEqual(result);
expect(getDeployment(result.deploymentId)?.sources).toEqual([{ kind: 'watchdog', source_id: 'incident', server }]); expect(getDeployment(result.deploymentId)?.sources).toEqual([{ kind: 'watchdog', source_id: 'incident', server,
label: 'Server down', href: '/maintenance/incident' }]);
expect(await operation(result.deploymentId)).toMatchObject({ triggered_by: 'watchdog:incident' }); expect(await operation(result.deploymentId)).toMatchObject({ triggered_by: 'watchdog:incident' });
db().prepare("INSERT INTO settings (key,value) VALUES ('watchdog_enabled','0')").run(); db().prepare("INSERT INTO settings (key,value) VALUES ('watchdog_enabled','0')").run();
await processProvisioning(); await processProvisioning();
+2 -1
View File
@@ -348,7 +348,8 @@ describe('Provisioning ownership', () => {
const source = db().prepare('SELECT provisioning_id FROM deployment_servers WHERE deployment_id = ?').get('campaign') as { provisioning_id: string }; const source = db().prepare('SELECT provisioning_id FROM deployment_servers WHERE deployment_id = ?').get('campaign') as { provisioning_id: string };
const id = source.provisioning_id; const id = source.provisioning_id;
expect(provisioningOperation(id)).toMatchObject({ action: 'restart', state: 'waiting', submitted_at: null }); expect(provisioningOperation(id)).toMatchObject({ action: 'restart', state: 'waiting', submitted_at: null });
expect(getDeployment(id)?.sources).toEqual([{ kind: 'campaign', source_id: 'campaign', server: servers[0] }]); expect(getDeployment(id)?.sources).toEqual([{ kind: 'campaign', source_id: 'campaign', server: servers[0],
label: 'Promotion campaign', href: '/campaigns/campaign' }]);
expect(komodoRestart).not.toHaveBeenCalled(); expect(komodoRestart).not.toHaveBeenCalled();
expect(komodoInspectHealthy).not.toHaveBeenCalled(); expect(komodoInspectHealthy).not.toHaveBeenCalled();
expect(rconGetPlayerCount).not.toHaveBeenCalled(); expect(rconGetPlayerCount).not.toHaveBeenCalled();