diff --git a/README.md b/README.md index 96ce9c0..6245920 100644 --- a/README.md +++ b/README.md @@ -433,7 +433,16 @@ distinguished. Provisioning (`/provisioning`, `/api/provisioning`) owns Create, Start, Restart, Rebuild, Stop, and Destroy: request persistence, eligibility and occupancy checks, execution, recovery, and history. Campaigns, betas, and scheduled maintenance request lifecycle work through Provisioning and link to the resulting operation. They retain their file, test-evidence, and version-verification workflows. Pending automated restarts appear in Provisioning before execution. History survives server removal. New creation requests return HTTP 202 with an operation ID and open the provisioning detail page. Creation is complete only when Komodo reports healthy and RCON or fresh telemetry verifies game status with an engine map that exactly matches the selected map. Queued requests reserve their server name and port; queued creation and pending health checks survive application restarts. The detail page records startup progress and explains any health or map mismatch that is still blocking completion. -Each server deployment opens a detail page with its flow and timestamped activity. Destroy uses an in-app confirmation modal and immediately returns a queued deployment (`DELETE /api/servers/:name` returns HTTP 202 with `deploymentId`). The worker waits for a fresh zero-player RCON or telemetry report, or a verified stopped container, before teardown. Occupied reports from either source block teardown; servers without current verification remain queued and are checked each worker tick. Waiting requests survive application restarts, while interrupted teardown is marked for inspection. Completed destruction stays on the provisioning detail page. +Each server deployment opens a detail page with its outcome, blocker, permitted +recovery actions, grouped execution phases, and timestamped verification evidence. +New jobs record command certainty and the source of game-health checks; older +records retain their known outcomes without invented evidence. A failed restart's +ongoing recovery and held capacity appear separately from its recorded failure. +Recent activity is visible immediately, with searchable history loaded in pages. +Launch files distinguish the setup captured while queued from the files selected +when creation began. The page retains its last snapshot if a refresh fails. + +Destroy uses an in-app confirmation modal and immediately returns a queued deployment (`DELETE /api/servers/:name` returns HTTP 202 with `deploymentId`). The worker waits for a fresh zero-player RCON or telemetry report, or a verified stopped container, before teardown. Occupied reports from either source block teardown; servers without current verification remain queued and are checked each worker tick. Waiting requests survive application restarts, while interrupted teardown is marked for inspection. Completed destruction stays on the provisioning detail page. Use **Provisioning → Manage server**, or a server’s **Server actions**, to start, diff --git a/docs/provisioning-implementation.md b/docs/provisioning-implementation.md new file mode 100644 index 0000000..e22ef59 --- /dev/null +++ b/docs/provisioning-implementation.md @@ -0,0 +1,44 @@ +# Provisioning detail implementation + +Implemented September 28, 2026, following the [approved review plan](provisioning-improvement-plan.md). + +## Delivered behavior + +The job page now leads with identity, recorded outcome, blocker, and permitted recovery action. Grouped execution phases and timestamped evidence precede supporting metadata and launch files. Mobile follows the same reading order. Operation IDs and links are copyable, and the return link preserves Provisioning filters or leads back to related work. + +The worker records versioned progress snapshots and semantic events at existing decision points. Container, game, map, and occupancy checks identify their source and observation time. Durable milestones preserve remote deletion, local cleanup, command certainty, and operation-owned recovery capacity. A failed restart can retain its failed outcome while later recovery evidence records capacity release. Legacy records retain their original stage and outcome without invented checkpoints. + +Cleanup resumes honor backend capabilities even for deleted identities. Confirmed remote deletion and the existing cleanup-resume stage are saved together before local cleanup, preserving recovery if a later guard or local operation fails. Late observations are rejected after cancellation, deletion, or replacement. Existing occupancy rules, scheduling, command counts, and observation-only resumes are covered by regression tests. + +The page uses server-loaded detail and abortable, visit-scoped refreshes. Action feedback survives background refresh failures; outdated requests cannot replace another job's detail. Terminal records continue visibility-aware polling because recovery, deletion, and capabilities can still change. + +Five recent events are visible. Full history loads in 25-entry pages with literal search, ordering, and a fixed upper event ID. Recent activity stays still while full history is open; new events appear through an explicit Jump to latest control. The legacy detail API still returns full events by default. The new page opts into `?events=recent`. + +Queued recipe captures and files selected for an attempt have separate labels and timestamps. Neither implies successful installation. Related work names resolve when available, while missing source IDs remain visible without broken links. + +## Validation + +- Full suite: **898 tests passed across 72 files**. +- Svelte/TypeScript: `npm run check` passed with **0 errors and 0 warnings**. +- Production: `npm run build` passed with the Node adapter. +- Database tests cover additive migration, historical identity and event preservation, transaction rollback, stable phase timing, semantic outcomes across same-ID resumes, and full versus bounded API compatibility. +- Lifecycle tests cover ambiguous and thrown remote replies, telemetry and map evidence, cancelled/replaced identity races, exact recovery ownership, and destroy/rebuild cleanup after confirmed deletion without repeating remote commands. +- Refresh/presentation tests cover navigation and action races, persistent acknowledged action results, failed reconciliation, cancellation, deleted cleanup, legacy evidence, and malformed responses. +- Firefox checks exercised history pagination, literal search, stable new-event handling, failed-refresh retention and recovery, and layouts at 1440, 390, and 320 pixels without page-wide overflow. Enter and Space operate phase disclosures. A deleted-identity cleanup resume completed through the real local API; retained restart recovery, cancellation, and legacy evidence also rendered correctly. +- Reflow also passed at 720 CSS pixels with pixel ratio 2, and long unbroken identifiers fit at 320px. This simulates the available layout width of 200% zoom on a 1440px display; native browser zoom was not measured. A separate CSS-zoom probe kept the detail content in bounds but exposed overflow in the existing global navigation. + +Browser fixtures use a temporary database and disabled external integrations. They exercise the real page and local detail/history APIs; lifecycle tests stub remote systems. This is not a production fleet smoke test or a formal accessibility conformance audit. The build reports the repository's existing `csrf.checkOrigin` deprecation notice. + +## Measurements and screenshots + +For a synthetic 5,001-event operation, full detail serialized to **1,884,119 bytes** and bounded detail to **3,011 bytes**. Thirty local bounded reads averaged **2.93 ms**. These measurements establish payload reduction on the local fixture, not a production latency guarantee; event totals and search still query retained history. + +In the 1440px create/map-wait fixture, the execution heading moved from approximately **1,073px** in the original composition to **675px**. The page keeps the outcome above execution and the recipe in a supporting column. Phone layouts move context below execution and activity. + +Updated screenshots: [desktop create/map wait](provisioning-review-assets/implemented-create-desktop.png), [mobile create/map wait](provisioning-review-assets/implemented-create-mobile.png), [deleted identity with permitted cleanup](provisioning-review-assets/implemented-deleted-cleanup.png), and [failed restart with retained recovery](provisioning-review-assets/implemented-restart-recovery.png). The original review captures remain linked from the approved plan. + +## Compatibility and rollout + +The schema additions are nullable, and old stage text and default API event responses remain supported. Progress is written as new work is evaluated; historical operations without structured evidence use the legacy presentation. Detail reads perform no remote probes. + +The change is implemented in the workspace. Normal deployment should verify one newly instrumented job and one legacy record against the configured integrations. No production deployment was performed during this work. diff --git a/docs/provisioning-improvement-plan.md b/docs/provisioning-improvement-plan.md new file mode 100644 index 0000000..ca8516b --- /dev/null +++ b/docs/provisioning-improvement-plan.md @@ -0,0 +1,230 @@ +# Provisioning job detail improvement plan + +Review date: September 28, 2026. Source baseline: `5b65520`. + +The recommendation is to make `/provisioning/:id` a complete operational record: identify the job, explain its current outcome and blocker, expose the evidence behind that outcome, and show the next permitted action. Progress and recovery should precede historical launch files. + +Approved and implemented on September 28, 2026. The original review and agreed scope remain below; see the [implementation and validation record](provisioning-implementation.md) for delivered behavior, checks, and updated screenshots. + +## Review method and participants + +The requested panel covered three UI/UX roles, two CI/CD systems engineering roles, and one software engineering role. The platform allowed only three child-agent threads and rejected additional spawns even after those reviews completed. The three threads therefore performed two separate rounds, first UX and then engineering. This provides six role-based reviews, rather than six independent agents. Each round included direct peer discussion; engineering reviewed and challenged the UX proposals. + +| Review role | Proposal and contribution | +| --- | --- | +| UI/UX: information architecture | Prioritize job identity, outcome, blocker, safe action, progress, activity, and broader work context. Separate historical completion from current server health. | +| UI/UX: interaction and accessibility | Use state-specific recovery actions, stable refresh behavior, persistent action feedback, keyboard-accessible progress, and usable history while updates arrive. | +| UI/UX: visual design | Replace the overloaded summary and wrapping step chips with a compact outcome card, grouped vertical progress, and supporting metadata. Reuse the existing design system. | +| CI/CD systems: lifecycle and queues | Preserve destructive checkpoints, occupancy gates, scheduling, command certainty, identity guards, and recovery-slot semantics. Make queue claims match actual admission behavior. | +| CI/CD systems: observability and reliability | Record the source and age of verification evidence, distinguish page freshness from worker checks, preserve recovery evidence, and validate compatibility and failure scenarios. | +| Software engineering | Define shared types and component boundaries, reuse server-loaded detail, fix route/poll/action races, and stage additive implementation with regression coverage. | + +The review covered the page, shared history component, lifecycle APIs, worker, queues, persistence, related collection/maintenance/campaign pages, and relevant tests. Root also inspected the current source in Firefox using an isolated temporary database, disabled integration credentials, and synthetic detail responses. Desktop captures used 1440px width; mobile used 390px. These are composed UI scenarios, not executions of real provisioning workflows or measurements of production use. + +Review assets: [create waiting for a map](provisioning-review-assets/create-map-wait-desktop.png), [mobile rebuild composition](provisioning-review-assets/rebuild-mobile.png), and [deleted identity with resumable cleanup](provisioning-review-assets/deleted-cleanup-desktop.png). + +## What feels incomplete today + +The page already has useful recovery buttons, server diagnostics links, a recorded launch recipe, source backlinks, elapsed time, progress steps, and searchable activity. Preserve those capabilities. The gaps concern their priority and the reliability of the story they tell. + +| Finding | Effect on the operator | Evidence | +| --- | --- | --- | +| One summary mixes status, actions, errors, warnings, diagnostics, map, timing, and provenance. The recipe appears before progress. | Monitoring and recovery compete with secondary information. | [Detail page](../src/routes/provisioning/[id]/+page.svelte), summary and recipe markup. In the synthetic desktop create case, the flow began about 1,073px down. | +| Rebuild renders 12 equally weighted wrapping step chips. | Removal, cleanup, and fresh creation do not read as distinct phases. | [Detail page](../src/routes/provisioning/[id]/+page.svelte), `steps`; [OperationHistory](../src/lib/ui/components/OperationHistory.svelte). | +| Progress and step duration depend on matching human-readable stage/event text. | New wording can move the highlighted step backward; changes in player count can appear to restart a step timer. | Detail page `currentStep` and `stageStarted`; [creation verification](../src/lib/server/provisioning/instances.ts) now emits “RCON or fresh telemetry” stages. | +| Runtime cancellation is handled after runtime-specific explanation logic. | A cancelled start/restart/stop can still claim automatic verification is underway. | Detail page `explanation`. | +| The deleted-server flag suppresses every cleanup button and the active-state presentation. | A permitted local cleanup can become inaccessible or appear inactive after resuming. | Detail page recovery controls and `active` versus [history.ts](../src/lib/server/provisioning/history.ts), `canResumeCleanup`. Missing button reproduced with a synthetic permitted response. | +| Completion copy asserts RCON health in the present tense. | Telemetry-based verification is misreported, and historical success can look like current health. | Detail page explanations; [status-evidence.ts](../src/lib/server/fleet/status-evidence.ts), `probeServerStatus`. | +| One refresh guard and one error string handle reads, mutations, and navigation. | A post-action reload can be dropped, an old request can affect another job, and polling can erase an action error. | Detail page `load`, `resume`, `cancelRemoval`; source-inferred race cases. | +| A failed restart can still hold a recovery slot. | A simple failed badge does not explain continued observation or effects on other jobs. | [restarts.ts](../src/lib/server/provisioning/restarts.ts), `refreshRestartRollout`. | +| Activity is collapsed, operation ID is absent from the page body, and related-work links are scattered. | Investigation and handoff require avoidable searching and navigation. | Detail page and shared history component. | + +## Proposed page + +Use one reading order across all six actions. At wide widths, metadata and launch inputs can occupy a supporting column. On phones they follow progress and activity in the DOM. Avoid a sticky sidebar. + +```text +Provisioning / Restart server +Origins · Lobby 1 t6_zombie_origins_1 +Operation ID [Copy] [Copy link] [View server] + +[Waiting] Waiting for players to leave +Last evaluated: 3 players · source and observation time +Continues automatically after an eligible empty/stopped check. +Relevant permitted action or dependency link +Total elapsed · Time in phase · Page updated / Refresh + +EXECUTION AND EVIDENCE OPERATION CONTEXT + Eligibility Requested by / requested at + Container action Submission attempted / finished + Readiness verification Related campaign/beta/maintenance + Container Collection launch + Game status + Selected map, when required Recorded launch files + [Launch another server] +RECENT ACTIVITY + Five latest entries + relevant failure + [Full searchable history] +``` + +The example evidence requires the new contract below; do not populate it from guessed values. + +### Identity, outcome, and next action + +- Show the action, friendly map/lobby title, technical server name, and copyable operation ID/link. Use existing identity and copy components. +- Lead with the specific blocker or outcome, followed by what happens next. Place the recorded error and permitted recovery control together. Give warnings a labeled explanation rather than color alone. +- Treat deletion as historical identity context. It must not replace the outcome, hide an actionable cleanup, or link an old operation to a replacement server with the same name. +- Determine active progress from the operation and recovery state, independently of whether a live server link is available. +- Distinguish a recorded outcome from ongoing recovery. A failed request can show “Recovery is still being checked; restart capacity remains reserved” when the backend records that condition. +- Use past-tense completion language with its timestamp: “Creation completed. Container readiness, game status, and the selected engine map were verified at …”. Name RCON or telemetry only when recorded. Current health remains on the server page. +- Preserve an explicit return link to the originating Provisioning filters or parent work, with a safe local fallback. Group all requesting campaigns, betas, maintenance records, and collection launches under Related work. Keep retained source IDs visible when their originating record no longer exists, without a broken link. +- Finishing a restart does not mean its campaign, beta, or maintenance verification has finished. Link back to that work with accurate wording. + +### Progress and evidence + +Use a vertical checklist with grouped phases, icons, explicit text states, and the current/failed phase expanded. Keep completed phases available for inspection. Avoid percentage complete or predicted finish times. + +| Action | Phase sequence and completion evidence | +| --- | --- | +| Create | Request/startup queue → prepare files and create deployment → start container → container readiness → game status → matching engine map. | +| Start | Eligibility and restart capacity → start existing container → container and game verification. Do not imply files were redeployed. | +| Restart | Eligibility, occupancy, and capacity → restart submission → container and game verification. Show any continuing recovery separately from request failure. | +| Rebuild | Eligibility/capacity → remove old container and deployment → local files/registry cleanup → queue fresh instance → create/start → readiness and map verification. Preserve the reserved identity/port and show when the old instance is gone. | +| Stop | Eligibility and occupancy → stop submission → explicit stopped-container confirmation. Explain that files and registration remain. | +| Destroy | Eligibility and occupancy → container removal → deployment deletion → local files/registry cleanup. Retain the historical record after removal. | + +Rebuild grouping must retain each irreversible checkpoint. “Remote deployment deleted” and “Local cleanup complete” are different facts; fresh creation must never look complete while old files remain. + +Progress supports waiting, current, completed, failed, cancelled/not run, skipped when actually recorded, and unknown. Do not mark earlier steps complete simply because a string matched a later label. Legacy records retain their raw stage and known outcome, with unrecorded milestones labeled accordingly. + +The blocker panel shows the worker's actual reason: occupied/unknown players, missing configuration, requester hold, conflicting work, startup or restart capacity, removal/port dependency, container readiness, game evidence, or wrong map. Include the blocking operation/source link when known. Keep “Requested by” separate from “Waiting on.” + +Show relevant capacity limits, holders, and this operation's reservation from the same backend rules. If a submission rank is shown, label it as request order: blocked earlier jobs can be bypassed, and source priorities affect admission. Do not say “You are next.” + +Evidence rows show result, evaluation time, source, and expected/observed values where relevant. Engine-map verification does not independently verify a T6 variant/start location. Unknown players are never displayed as zero. + +Separate durable milestones from transient readiness checks. A passed container/game/map observation remains evidence of what passed at its timestamp; a later evaluation may make the current check failed or unknown. Do not leave stale green checks implying current readiness. + +### Actions and terminal states + +| Backend condition | Page behavior | +| --- | --- | +| `canCancelRemoval` | Offer **Cancel queued removal**, explaining that it is available only before removal begins. Preserve the existing atomic worker/cancellation race protection. | +| `canResumeVerification` | Offer **Resume health verification** and explain that it observes the existing instance without creating or restarting it. | +| `canResumeCleanup` | Offer **Resume cleanup**, including eligible deleted identities. Explain that Komodo deletion already succeeded. | +| `canResumeRebuild` | Offer **Resume rebuild cleanup** and explain that local cleanup precedes creation in the reserved slot. | +| Failed/interrupted without a resume capability | Lead with the recorded failure and relevant inspection destination. An uncertain remote response must not produce a generic retry command. | +| Cancelled | State that the request is cancelled; distinguish recorded prior work from remaining work that did not run. Do not claim continued execution. | +| Warning | Explain the specific remaining issue. Do not assume every warning concerns cleanup. | +| Complete | Show the verified historical outcome and appropriate View server / Back to related work / Back to history link. | + +The API revalidates every action. A 409 triggers reconciliation of current capabilities and explains the state change; it never automatically repeats a mutation. Disable duplicate submissions while an action is pending, keep its accessible name, and preserve its result independently of background refresh. If an acknowledged successful action is followed by a failed refresh, retain its success acknowledgment alongside stale-data feedback; a refresh failure does not turn the action into a failure. + +“Launch another server” remains secondary beside the recipe, with its existing explanation that it creates a separate server using current promoted files. It is not a repair or replay of the recorded launch. Create and rebuild already save a recipe while queued, then recapture promoted files when creation starts. Label these **Setup captured when queued** versus **Files selected for this attempt**, with capture times and an explicit recipe status. The queued capture is not a version pin, and selected files do not prove installation succeeded. Legacy records without that distinction retain neutral recorded-recipe wording. Where no recipe exists, explain the absence. + +### Activity, refresh, and accessibility + +- Keep five recent events visible and show a targeted failure excerpt when needed. Retain full-history search, ordering, and Show more. Do not automatically open a large history when a failure arrives. +- Load full history on demand in bounded pages rather than retransmitting the entire retained history every five seconds. Preserve the legacy API's default response through an opt-in page query. +- Preserve search, disclosure, focus, and reading position across updates. While inspecting older history, show a new-event count and **Jump to latest** instead of moving the content underneath the reader. +- Separate initial loading failure, background refresh failure, and action failure. Keep last-known detail visible with an explicit stale message and manual Refresh. +- Show **Page updated** separately from **Last evaluated** and evidence receipt time. A successful GET does not prove the worker or server was checked recently. An unchanged event stream does not prove the worker is stalled. +- Keep the existing five-second visibility-aware polling initially, including terminal records. Failed jobs may still have recovery activity; another tab may resume them; later deletion can change identity links and capabilities. +- Use server-loaded initial detail. On route changes and actions, use abortable reads and request generations so old responses cannot replace new detail or feedback. Force reconciliation after an action even if a poll was already in progress. +- Announce meaningful outcome/action changes politely, not every timer tick. Keep textual progress states and `aria-current`; errors remain near their actions. Use a stable loading layout and a clear missing-operation state. +- Reuse existing primitives, spacing, colors, typography, and identity components. Verify long identifiers, keyboard access, visible focus, contrast, reduced motion, 320/390px layouts, tablet/desktop layouts, and 200% zoom. + +## Data and implementation design + +The page needs structured facts that the current response does not fully expose. Add a small typed read model and observation metadata; preserve the existing lifecycle state machine and execution predicates. + +| Contract area | Existing foundation | Required addition | +| --- | --- | --- | +| Identity and outcome | `server_deployments`, historical identity guards | Shared typed detail; recorded result remains distinct from current recovery. | +| Phase and timing | Free-text `stage`, request/finish/submission times | Nullable stable phase key/start time; explicit checkpoint states/times; total elapsed remains based on the original request. | +| Blocker | Worker decision branches and stage messages | Reason code, explanation, evaluated time, automatic continuation condition, and exact dependency reference where known. | +| Command certainty | `submitted_at` is persisted before the network call | Distinguish submission attempted, reply accepted, uncertain outcome, and verified result. Never relabel submission intent as remote acceptance. | +| Verification | Existing probes and transient results | Versioned snapshot with container/game/map/occupancy evidence, result/source/evaluation time, source receipt time where available, and expected/actual values. | +| Recovery and capacity | `restart_rollout`, creation queue, saved concurrency | Exact operation-owned slot/reservation status, relevant limits/holders, and durable recovery/release evidence. | +| Available actions | Existing `canResume*` and `canCancelRemoval` checks | Keep these authoritative; supply contextual explanation without creating new execution capabilities. | +| Provenance and recipe | Sources, collection, saved launch recipe | Named/resolvable context links, missing/deleted-source handling, and explicit queued-capture versus attempted-selection status. | +| Activity | Ordered event ID/time/message | Optional semantic kind/phase/details while preserving existing event text and order. | + +A suitable small schema extension is nullable `phase_key`, `phase_started_at`, `observed_at`, and versioned `observation_json` on the operation, plus optional event `kind`, `phase_key`, and allowlisted `details_json`. Final naming can follow repository conventions. Snapshot payloads remain bounded and contain only the fields needed by the page, not raw integration responses or credentials. + +Implementation rules: + +1. Record structured blockers and observations where the worker already makes decisions. Do not build a second eligibility engine in the API or browser. The detail GET remains read-only and performs no Komodo/RCON probes. +2. Derive current capacity from existing scheduler helpers/tables, with an explicit snapshot time. Persist important milestones against the operation because server-owned recovery rows can later be replaced or deleted. Only attach a live slot when its `operation_id` matches; slot absence alone is not evidence of successful release. +3. Update related state, observation, and semantic event atomically around each durable checkpoint, keeping network calls outside database transactions. Dual-write existing stage/event text, especially `REMOVING_SERVER`, which participates in recovery eligibility today. +4. Record meaningful phase, blocker, command, and recovery transitions. Update observation timestamps on repeated checks without appending an event every poll. Changing player counts must not reset the phase's start time. +5. Preserve prior failure and resume events when an existing operation resumes. New observations must not rewrite the earlier command result or borrow evidence from a newer instance. This does not introduce a new retry engine. +6. For historical records, expose available durable facts and unknowns. Do not backfill invented success timestamps, evidence sources, or completed phases from ambiguous prose. +7. Keep existing IDs, history, API properties, `/api/deployments` compatibility, and legacy redirects. Use additive migrations and test both fresh and upgraded databases. Preserve the legacy `deployment.warnings` string; expose a validated string array through the additive page projection, with an honest fallback for malformed historical values. + +Suggested boundaries: + +- Shared provisioning detail/progress types near [provisioning.ts](../src/lib/provisioning.ts); a pure presentation mapper for labels, action/state coverage, and legacy fallbacks. +- Server serialization and persistence helpers near [history.ts](../src/lib/server/provisioning/history.ts), with instrumentation in [worker.ts](../src/lib/server/provisioning/worker.ts), [instances.ts](../src/lib/server/provisioning/instances.ts), and [restarts.ts](../src/lib/server/provisioning/restarts.ts). +- An abortable provisioning refresh/action controller under `src/lib/ui/`, taking lessons from the existing metrics refresh controller without changing `pollVisible` globally. +- A thin detail route composed from outcome/actions, execution/evidence, and supporting context. Make activity improvements opt-in or provisioning-specific so campaign/beta history defaults stay intact. +- Reuse the already-fetched detail in [the server loader](../src/routes/provisioning/[id]/+page.server.ts), preserving its campaign fallback and 404 behavior. + +The current API returns all events on every poll; the existing Show more limits rendering only. Add an opt-in bounded detail query for this page and its initial server load: five recent entries, event total/latest ID, and the relevant failure excerpt. Fetch full history through `/api/provisioning/:id/events` in 25-entry pages with stable event-ID cursors, literal-text search, and explicit order. Keep existing default detail responses and `/api/deployments` behavior compatible. New events must not shift the active history page; show their count until the reader requests them. Test ordering and cursor stability with concurrent inserts, and measure representative long histories. This is a bounded query change, not a new event-storage system. + +## Discussion decisions + +| Proposal or disagreement | Decision and reason | +| --- | --- | +| Add a richer dashboard versus reorganize current capabilities | Reorganize around one operation and add only the evidence needed to explain it. Existing recipes, diagnostics, and recovery controls remain useful. | +| Automatically expand all history on failure | Show the failure excerpt and five recent events; keep full expansion under operator control to avoid long, moving pages. | +| Continue fetching all events because only five are displayed | Use an opt-in bounded detail/history API. A five-entry visual preview alone does not bound network or database work, and provisioning history is retained. Preserve legacy consumers. | +| Compress rebuild into a few visual phases | Accepted only if remote deletion, local cleanup, replacement queueing, and verification remain separately inspectable. | +| Improve progress using more stage-string matching | Rejected as the primary model. Use typed worker facts with an honest legacy fallback. Preserve old strings for compatibility and safety predicates. | +| Show an exact queue position or ETA | Show actual blockers/capacity and optionally request order. Admission skips blocked requests; durations are not predictable from rank. | +| Stop refreshing completed/failed jobs | Keep visibility-aware polling initially because recovery, deletion, and permitted actions can still change. | +| Treat Deleted as making every action unavailable | Use backend capabilities. A historical server can still have permitted local cleanup without a live-server link. | +| Treat failed request and recovery as one status | Keep them separate: the historical failure may remain while recovery continues and releases capacity. | +| Add a new event system or scheduler | Rejected. Use additive snapshots and semantic metadata at existing execution boundaries. | + +## Delivery sequence + +These four increments form the proposed implementation scope. The data contract can be agreed first while layout work proceeds against fixtures. + +| Increment | Work | Reviewable result | +| --- | --- | --- | +| 1. Trustworthy operational facts | Shared types, additive migration, worker checkpoint/evidence writes, typed read model, capability and legacy handling. | All six actions provide accurate progress and unknowns without changing execution, command counts, occupancy rules, or slot release. | +| 2. Correct state and refresh behavior | Replace string-derived presentation; fix cancelled/runtime and telemetry copy, deleted cleanup gating, historical outcomes; reuse server-loaded detail; isolate navigation/poll/action requests and feedback. | The existing page remains truthful during cancellation, resume, slow requests, offline refresh, navigation, and continued recovery after failure. | +| 3. Complete detail-page composition | Outcome/action card, blocker/evidence panel, grouped vertical phases, five recent events, bounded full-history queries, metadata/provenance/recipe placement, copy links, return context, responsive/accessibility work. | Operators can find the job's outcome, blocking condition, and valid next action without searching the activity log. | +| 4. Compatibility and release validation | Exercise the full scenario matrix, migration/API compatibility, shared-component regressions, browser behavior, docs, and long-history measurements. | Implementation is ready for normal deployment with documented semantics and evidence of preserved lifecycle guarantees. | + +Dependencies: evidence-dependent controls wait for increment 1; refresh and visual work can run in parallel against the agreed contract. Increment 4 is a release gate, not a later cleanup task. + +## Acceptance and verification + +The review baseline passed **84 tests across five suites**: provisioning, creation queue, operation history, deleted-server guards, and restart rollout. Those tests establish the existing baseline; they do not validate the proposed implementation. No full-build or accessibility-conformance claim is made for this planning work. + +Implementation acceptance: + +| Scenario | Required result | +| --- | --- | +| Every action across applicable queued/waiting/running/verifying/success/warning/failure/interruption/cancellation states | Correct headline, phase, evidence, timing, and permitted next action. Impossible combinations have explicit legacy/unknown handling. | +| Occupied, unknown, stopped, stale, conflicting RCON/telemetry evidence | UI matches existing eligibility decisions; positive occupancy blocks disruption; stale/pre-submission/future telemetry cannot masquerade as accepted evidence. | +| Capacity or parent/dependency hold | Actual blocker and matching links appear; no guaranteed admission order, fabricated check time, or ETA. | +| Ambiguous remote reply, then recovery | No duplicate external command; slot stays reserved until the existing rules release/transfer it; historical failure and recovery evidence remain distinguishable. | +| Partial destroy/rebuild and deleted identity | Confirmed remote deletion is retained; local cleanup can resume only when permitted; creation waits for full cleanup; old work never targets a replacement instance. | +| Cancellation or resume racing a worker claim/newer operation | Backend remains authoritative; 409 reconciles the page; no automatic mutation replay or false success. | +| Resume of the same operation ID | Prior failure outcome/time and explicit resume event remain preserved in semantic history while current progress/finish fields update; later completion records its own outcome without a new retry system. | +| Telemetry success / wrong map / historical success | Evidence source and timestamps are accurate; expected and observed engine maps are clear; the page does not claim current health from old results. | +| Slow poll + action, A→B→A navigation, stale error, unmount | Obsolete responses cannot affect current detail or action feedback; successful mutations invalidate stale reads and trigger reconciliation; pending state resets appropriately. If reconciliation fails, action acknowledgment and stale-data feedback remain distinct. | +| Refresh failure and recovery | Last snapshot remains visible with stale labeling; Retry/Refresh works; action errors are not silently erased. | +| Long activity, new events while reading | Search/order/disclosures/focus remain stable; new-event indicator works; no forced scrolling. Responses stay bounded; cursor pages have no duplicates/gaps within their documented ordering; legacy defaults remain compatible. Measure payload/query/render cost. | +| Promotion occurs while create/rebuild is queued | Queued recipe is labeled as a capture, final attempted selection is recorded at creation, and the page does not imply pinned versions or successful installation. | +| Legacy DB, absent recipe/evidence/source, deleted/reused name | Honest missing-data presentation, preserved IDs/events/foreign keys, safe links/capabilities, and unchanged legacy route/API compatibility. | +| Keyboard, assistive announcements, phone, zoom, long text | No loss of reachable actions or page-wide overflow; progress is understandable without color; timers do not cause repeated announcements. | + +Extend behavior tests around these risks rather than snapshotting markup or mirroring DTO construction. Verify that detail GETs perform no external probes. Run the affected lifecycle/history/refresh tests, then the full suite, `npm run check`, and `npm run build` for the implementation. Validate desktop/mobile browser fixtures and update [Provisioning documentation](provisioning.md) and relevant README descriptions. + +Preserve the existing execution rules throughout: fresh occupancy gates, startup/restart limits, ambiguous-response handling, historical identity checks, exact cleanup checkpoints, requester ownership, and observation-only resumes. Normal deployment should sample both a newly instrumented operation and a legacy record; presentation rollback must not require deleting the new history. + +Outside this proposal: generic retry/force-run controls, new cancellation semantics, scheduler redesign, real-time transport migration, a general observability platform, broad redesigns of other operation pages, and predicted completion times. diff --git a/docs/provisioning-review-assets/create-map-wait-desktop.png b/docs/provisioning-review-assets/create-map-wait-desktop.png new file mode 100644 index 0000000..9c1e1cc Binary files /dev/null and b/docs/provisioning-review-assets/create-map-wait-desktop.png differ diff --git a/docs/provisioning-review-assets/deleted-cleanup-desktop.png b/docs/provisioning-review-assets/deleted-cleanup-desktop.png new file mode 100644 index 0000000..7f5cba0 Binary files /dev/null and b/docs/provisioning-review-assets/deleted-cleanup-desktop.png differ diff --git a/docs/provisioning-review-assets/implemented-create-desktop.png b/docs/provisioning-review-assets/implemented-create-desktop.png new file mode 100644 index 0000000..1d97f8d Binary files /dev/null and b/docs/provisioning-review-assets/implemented-create-desktop.png differ diff --git a/docs/provisioning-review-assets/implemented-create-mobile.png b/docs/provisioning-review-assets/implemented-create-mobile.png new file mode 100644 index 0000000..ac15d4d Binary files /dev/null and b/docs/provisioning-review-assets/implemented-create-mobile.png differ diff --git a/docs/provisioning-review-assets/implemented-deleted-cleanup.png b/docs/provisioning-review-assets/implemented-deleted-cleanup.png new file mode 100644 index 0000000..712242b Binary files /dev/null and b/docs/provisioning-review-assets/implemented-deleted-cleanup.png differ diff --git a/docs/provisioning-review-assets/implemented-restart-recovery.png b/docs/provisioning-review-assets/implemented-restart-recovery.png new file mode 100644 index 0000000..f081112 Binary files /dev/null and b/docs/provisioning-review-assets/implemented-restart-recovery.png differ diff --git a/docs/provisioning-review-assets/rebuild-mobile.png b/docs/provisioning-review-assets/rebuild-mobile.png new file mode 100644 index 0000000..eef4091 Binary files /dev/null and b/docs/provisioning-review-assets/rebuild-mobile.png differ diff --git a/docs/provisioning.md b/docs/provisioning.md index ee01193..d6b0e29 100644 --- a/docs/provisioning.md +++ b/docs/provisioning.md @@ -75,6 +75,59 @@ new attempt, preserving prior history; paused or closed beta work cannot start a queued restart. Recovery of an already submitted restart continues even if its requester closes. Upgrades adopt existing restart intent for observation only. +## Operation details + +Each job page puts its recorded outcome, blocking condition, and available next +action before launch files. Execution is grouped into phases, with separate +milestones for remote deletion, local cleanup, and replacement creation during a +rebuild. Copy the operation ID or link when investigating or handing off a job. +Related work links back to the requesting campaign, beta, maintenance item, or +collection. A completed provisioning job does not imply that its requester has +finished its own verification. + +New operations record structured verification evidence alongside their activity: +the check result, source, evaluation time, and expected/observed map when relevant. +Game status can be verified by RCON or accepted fresh telemetry. Engine-map checks +do not independently verify a T6 start location. Passed checks describe their +recorded observation; later checks can fail or become unknown. Completion describes +what was verified at that time, while **View server** opens the current instance. +Older history retains its recorded stages and outcomes; unavailable evidence is +not inferred from stage wording. + +Submission attempted, command accepted, and verified completion are separate +milestones. A failed restart may still have ongoing recovery observation and hold +restart capacity. Its original failure remains in history even if recovery later +succeeds and releases the slot. Capacity and request order are snapshots, not +guarantees of the next job admitted or an estimated completion time. Blocked +requests can be bypassed by eligible work. + +Recovery controls use the server's current eligibility checks. **Resume health +verification** observes the existing instance without submitting another create or +restart. **Resume cleanup** retries local cleanup after confirmed remote deletion, +including eligible historical deleted identities; replacement or newer-operation +guards still apply. **Resume rebuild cleanup** completes cleanup before creating +the replacement in its reserved slot. Resuming the same operation preserves its +earlier failure and resume events. Uncertain commands have no generic Retry action. + +Create and rebuild capture a setup when queued and select the latest promoted +files again when creation begins. The page distinguishes **Setup captured when +queued** from **Files selected for this attempt**. A queued capture does not pin +versions; an attempted selection is not proof that every file was installed. +**Launch another server** uses the latest promoted setup with a fresh identity and +port, rather than repairing or replaying the recorded job. + +Five recent events remain visible. Expand full history to search, change order, +and load older pages. A fixed history watermark keeps the viewed pages stable as +events arrive; **Jump to latest** incorporates new activity. The detail view +fetches a bounded recent-event window, while full history loads on demand. + +The page refreshes every five seconds while visible, including terminal jobs whose +recovery or available actions may still change. **Page updated** describes the +last successful detail fetch; it is separate from the worker's evaluation and +telemetry receipt times. A refresh failure keeps the last snapshot visible and +does not erase an acknowledged action result. These timestamps are not worker +heartbeat or completion-time promises. + ## Routes and compatibility - `/provisioning`: active operations and history filtered by action. @@ -89,6 +142,16 @@ requester closes. Upgrades adopt existing restart intent for observation only. - `/api/provisioning/:id`, `/:id/resume`, `/:id/cancel`, `/preview`, and `/house-party` preserve the existing operation, recovery, removal cancellation, launch preview, and fleet destruction APIs. +- `GET /api/provisioning/:id?events=recent` returns the same detail with five recent + events plus activity totals/latest ID and a failure excerpt. Omitting the option + preserves the full legacy event response. Progress, capacity, named related work, + and normalized warning messages are additive fields; the original warning string + and lifecycle fields remain available. +- `GET /api/provisioning/:id/events` reads history with `order=newest|oldest`, + literal-text `query`, `limit` (default 25, maximum 100), an exclusive event-ID + `cursor`, and an upper-ID `watermark`. Keep the returned watermark when loading + more and use `nextCursor` until it is null. Begin without a watermark to include + newly arrived events. These reads do not probe Komodo or RCON. Old `/deployments` pages redirect to their `/provisioning` equivalents, preserving query parameters. Collection launch links at `/provisioning/collections/:id` or diff --git a/src/lib/provisioning-detail.ts b/src/lib/provisioning-detail.ts new file mode 100644 index 0000000..70a5dbf --- /dev/null +++ b/src/lib/provisioning-detail.ts @@ -0,0 +1,121 @@ +import type { ProvisioningAction } from './provisioning.js'; +import type { LaunchRecipeSummary } from './deployment-plan.js'; + +export const provisioningPhases = ['requested', 'eligibility', 'startup-queue', 'prepare', 'create-deployment', + 'container-action', 'remove-container', 'delete-deployment', 'cleanup', 'verify-container', 'verify-game', + 'verify-map', 'verify-stopped'] as const; +export type ProvisioningPhase = typeof provisioningPhases[number]; +export type EvidenceResult = 'pass' | 'waiting' | 'fail' | 'unknown'; +export type EvidenceSource = 'komodo' | 'rcon' | 'telemetry' | 'combined' | null; + +export interface ProvisioningCheck { + key: 'container' | 'game' | 'map' | 'occupancy' | 'stopped'; + label: string; + result: EvidenceResult; + source: EvidenceSource; + checkedAt: string; + receivedAt: string | null; + expected: string | number | null; + actual: string | number | null; + message: string | null; +} +export interface ProvisioningBlocker { + code: string; + message: string; + automatic: boolean; + href?: string | null; + label?: string | null; +} +export interface ProvisioningCheckpoint { + key: string; + label: string; + at: string; + phase: ProvisioningPhase | null; +} +export interface RecoveryObservation { + state: 'held' | 'released' | 'transferred' | 'unknown'; + at: string; + message: string; + operationId?: string | null; +} +export interface ProvisioningProgress { + version: 1; + phase: ProvisioningPhase | null; + phaseStartedAt: string | null; + observedAt: string | null; + blocker: ProvisioningBlocker | null; + checks: ProvisioningCheck[]; + checkpoints: ProvisioningCheckpoint[]; + command: { state: 'attempted' | 'accepted' | 'rejected' | 'uncertain'; at: string; message: string } | null; + recovery: RecoveryObservation | null; + recipeStatus: 'queued' | 'attempted' | null; +} +export interface ProvisioningEvent { + id: number; + created_at: string; + message: string; + kind?: string | null; + phase_key?: string | null; + details_json?: string | null; +} +export interface ProvisioningDeployment { + id: string; + server: string; + server_deleted_at: string | null; + action: ProvisioningAction; + state: string; + stage: string; + started_at: string; + finished_at: string | null; + submitted_at?: string | null; + checked_at?: string | null; + players?: number | null; + triggered_by: string; + error: string | null; + warnings: string; + server_exists: number; + expected_map?: string | null; +} +export interface ProvisioningSource { + kind: 'campaign' | 'beta' | 'maintenance' | 'watchdog'; + source_id: string; + server?: string; + label: string; + href: string | null; +} +export interface ProvisioningCapacity { + evaluatedAt: string; + pools: { + kind: 'startup' | 'restart'; + limit: number; + used: number; + holders: { id: string | null; server: string }[]; + requestOrder: number | null; + ownsSlot: boolean; + }[]; +} +export interface ProvisioningDetail { + deployment: ProvisioningDeployment; + events: ProvisioningEvent[]; + activity: { total: number; latestId: number | null; failureExcerpt: ProvisioningEvent | null }; + sources: ProvisioningSource[]; + recipe: LaunchRecipeSummary | null; + blueprintName: string | null; + canResumeVerification: boolean; + canResumeCleanup: boolean; + canResumeRebuild: boolean; + canCancelRemoval: boolean; + collection: { id: string; name: string } | null; + progress: ProvisioningProgress | null; + capacity: ProvisioningCapacity; + warningMessages: string[]; + generatedAt: string; +} +export interface ProvisioningEventPage { + events: ProvisioningEvent[]; + total: number; + totalEvents: number; + latestId: number | null; + watermark: number; + nextCursor: number | null; +} diff --git a/src/lib/server/db/index.ts b/src/lib/server/db/index.ts index f5b62d5..eb49db0 100644 --- a/src/lib/server/db/index.ts +++ b/src/lib/server/db/index.ts @@ -243,8 +243,10 @@ function migrateProvisioningActions(handle: Database.Database): void { handle.transaction(() => { handle.exec(table.sql.replace('server_deployments', 'server_deployments_new') .replace("'create', 'destroy'", "'create', 'start', 'restart', 'rebuild', 'stop', 'destroy'")); - handle.exec(`INSERT INTO server_deployments_new (rowid, id, server, action, state, stage, started_at, finished_at, triggered_by, error, warnings) - SELECT rowid, id, server, action, state, stage, started_at, finished_at, triggered_by, error, warnings FROM server_deployments; + const columns = (handle.prepare('PRAGMA table_info(server_deployments)').all() as { name: string }[]) + .map(column => `"${column.name.replaceAll('"', '""')}"`).join(', '); + handle.exec(`INSERT INTO server_deployments_new (rowid, ${columns}) + SELECT rowid, ${columns} FROM server_deployments; DROP TABLE server_deployments; ALTER TABLE server_deployments_new RENAME TO server_deployments;`); if ((handle.pragma('foreign_key_check') as unknown[]).length) throw new Error('Provisioning migration found broken references'); @@ -309,7 +311,16 @@ export function db(): Database.Database { migrateTable(handle, 'server_deployments', [ { name: 'submitted_at', ddl: 'ALTER TABLE server_deployments ADD COLUMN submitted_at TEXT' }, { name: 'checked_at', ddl: 'ALTER TABLE server_deployments ADD COLUMN checked_at TEXT' }, - { name: 'players', ddl: 'ALTER TABLE server_deployments ADD COLUMN players INTEGER' } + { name: 'players', ddl: 'ALTER TABLE server_deployments ADD COLUMN players INTEGER' }, + { name: 'phase_key', ddl: 'ALTER TABLE server_deployments ADD COLUMN phase_key TEXT' }, + { name: 'phase_started_at', ddl: 'ALTER TABLE server_deployments ADD COLUMN phase_started_at TEXT' }, + { name: 'observed_at', ddl: 'ALTER TABLE server_deployments ADD COLUMN observed_at TEXT' }, + { name: 'observation_json', ddl: 'ALTER TABLE server_deployments ADD COLUMN observation_json TEXT' } + ]); + migrateTable(handle, 'server_deployment_events', [ + { name: 'kind', ddl: 'ALTER TABLE server_deployment_events ADD COLUMN kind TEXT' }, + { name: 'phase_key', ddl: 'ALTER TABLE server_deployment_events ADD COLUMN phase_key TEXT' }, + { name: 'details_json', ddl: 'ALTER TABLE server_deployment_events ADD COLUMN details_json TEXT' } ]); for (const table of ['deployment_servers', 'beta_servers', 'maintenance_jobs']) { migrateTable(handle, table, [{ name: 'provisioning_id', ddl: `ALTER TABLE ${table} ADD COLUMN provisioning_id TEXT` }]); diff --git a/src/lib/server/db/schema.sql b/src/lib/server/db/schema.sql index 9c1828e..80b5f95 100644 --- a/src/lib/server/db/schema.sql +++ b/src/lib/server/db/schema.sql @@ -452,14 +452,21 @@ CREATE TABLE IF NOT EXISTS server_deployments ( finished_at TEXT, triggered_by TEXT NOT NULL, error TEXT, - warnings TEXT NOT NULL DEFAULT '[]' + warnings TEXT NOT NULL DEFAULT '[]', + phase_key TEXT, + phase_started_at TEXT, + observed_at TEXT, + observation_json TEXT ); CREATE TABLE IF NOT EXISTS server_deployment_events ( id INTEGER PRIMARY KEY AUTOINCREMENT, deployment_id TEXT NOT NULL REFERENCES server_deployments(id) ON DELETE CASCADE, created_at TEXT NOT NULL, - message TEXT NOT NULL + message TEXT NOT NULL, + kind TEXT, + phase_key TEXT, + details_json TEXT ); CREATE INDEX IF NOT EXISTS idx_server_deployment_events ON server_deployment_events(deployment_id, id); diff --git a/src/lib/server/fleet/status-evidence.ts b/src/lib/server/fleet/status-evidence.ts index 545ef70..66ff843 100644 --- a/src/lib/server/fleet/status-evidence.ts +++ b/src/lib/server/fleet/status-evidence.ts @@ -36,30 +36,48 @@ export function telemetryEvidence(server: string, after: string | null = latestG /** Positive occupancy from either source always vetoes an empty-server claim. */ export async function probePlayerCount(server: string): Promise { + return (await probePlayerEvidence(server)).players; +} + +export interface PlayerEvidence { + players: number | null; + source: 'rcon' | 'telemetry' | null; + observedAt: string; + receivedAt?: string; +} + +/** The same eligibility decision, with provenance from the probes already performed. */ +export async function probePlayerEvidence(server: string): Promise { + const fromTelemetry = (value: TelemetryEvidence): PlayerEvidence => ({ + players: value.players, source: 'telemetry', observedAt: new Date().toISOString(), receivedAt: value.receivedAt + }); const occupied = telemetryEvidence(server); - if (occupied && occupied.players > 0) return occupied.players; + if (occupied && occupied.players > 0) return fromTelemetry(occupied); const players = await rconGetPlayerCount(server).catch(() => null); // Read after the network request: a player may join while RCON is pending. const telemetry = telemetryEvidence(server); - if (players !== null && players > 0) return players; - if (telemetry && telemetry.players > 0) return telemetry.players; - return players === 0 || telemetry?.players === 0 ? 0 : null; + if (players !== null && players > 0) return { players, source: 'rcon', observedAt: new Date().toISOString() }; + if (telemetry && telemetry.players > 0) return fromTelemetry(telemetry); + if (players === 0) return { players, source: 'rcon', observedAt: new Date().toISOString() }; + if (telemetry?.players === 0) return fromTelemetry(telemetry); + return { players: null, source: null, observedAt: new Date().toISOString() }; } /** Komodo readiness remains a caller prerequisite. Old-process telemetry is excluded. */ export async function probeServerStatus(server: string, after?: string | null): Promise< - { ok: true; players: number; map: string | null; source: 'rcon' | 'telemetry' } | { ok: false; reason: string } + { ok: true; players: number; map: string | null; source: 'rcon' | 'telemetry'; observedAt: string; receivedAt?: string } | { ok: false; reason: string; observedAt: string } > { const current = telemetryEvidence(server, after); - if (current) return { ok: true, ...current, source: 'telemetry' }; + if (current) return { ok: true, ...current, source: 'telemetry', observedAt: new Date().toISOString() }; let reason = 'RCON unavailable'; try { const rcon = await rconStatusResult(server); const latest = telemetryEvidence(server, after); - if (latest) return { ok: true, ...latest, source: 'telemetry' }; - if (rcon.ok) return { ok: true, players: rcon.status.players, map: rcon.status.map, source: 'rcon' }; + if (latest) return { ok: true, ...latest, source: 'telemetry', observedAt: new Date().toISOString() }; + if (rcon.ok) return { ok: true, players: rcon.status.players, map: rcon.status.map, source: 'rcon', observedAt: new Date().toISOString() }; reason = rcon.reason; } catch (err) { reason = (err as Error).message; } const telemetry = telemetryEvidence(server, after); - return telemetry ? { ok: true, ...telemetry, source: 'telemetry' } : { ok: false, reason }; + return telemetry ? { ok: true, ...telemetry, source: 'telemetry', observedAt: new Date().toISOString() } + : { ok: false, reason, observedAt: new Date().toISOString() }; } diff --git a/src/lib/server/provisioning/activity.ts b/src/lib/server/provisioning/activity.ts new file mode 100644 index 0000000..c9ad121 --- /dev/null +++ b/src/lib/server/provisioning/activity.ts @@ -0,0 +1,56 @@ +import { db } from '../db/index.js'; +import type { ProvisioningEvent, ProvisioningEventPage } from '../../provisioning-detail.js'; + +type ActivityRow = ProvisioningEvent; + +export interface ActivityQuery { + order?: 'newest' | 'oldest'; + query?: string; + cursor?: number; + watermark?: number; + limit?: number; +} + +export function activitySummary(id: string): { total: number; latestId: number | null } { + return db().prepare(`SELECT COUNT(*) AS total, MAX(id) AS latestId + FROM server_deployment_events WHERE deployment_id = ?`).get(id) as { total: number; latestId: number | null }; +} + +/** ID ordering and a fixed upper watermark keep older pages stable as new events arrive. */ +export function deploymentActivity(id: string, options: ActivityQuery = {}): ProvisioningEventPage { + const summary = activitySummary(id); + const watermark = Math.min(options.watermark ?? summary.latestId ?? 0, summary.latestId ?? 0); + const limit = Math.max(1, Math.min(100, Math.trunc(options.limit ?? 25))); + const newest = options.order !== 'oldest'; + const query = options.query ?? ''; + const base = 'deployment_id = ? AND id <= ? AND instr(lower(message), lower(?)) > 0'; + const values: (string | number)[] = [id, watermark, query]; + const total = (db().prepare(`SELECT COUNT(*) AS n FROM server_deployment_events WHERE ${base}`) + .get(...values) as { n: number }).n; + const cursor = options.cursor === undefined ? '' : ` AND id ${newest ? '<' : '>'} ?`; + if (options.cursor !== undefined) values.push(options.cursor); + const rows = db().prepare(`SELECT id, created_at, message, kind, phase_key, details_json + FROM server_deployment_events WHERE ${base}${cursor} + ORDER BY id ${newest ? 'DESC' : 'ASC'} LIMIT ?`).all(...values, limit + 1) as ActivityRow[]; + const more = rows.length > limit; + const events = rows.slice(0, limit); + return { events, total, totalEvents: summary.total, latestId: summary.latestId, watermark, + nextCursor: more ? events.at(-1)!.id : null }; +} + +export function deploymentEvents(id: string, recent = false): ProvisioningEvent[] { + if (recent) return (db().prepare(`SELECT id, created_at, message, kind, phase_key, details_json + FROM server_deployment_events WHERE deployment_id = ? ORDER BY id DESC LIMIT 5`).all(id) as ActivityRow[]).reverse(); + return (db().prepare(`SELECT id, created_at, message, kind, phase_key, details_json + FROM server_deployment_events WHERE deployment_id = ? ORDER BY id`).all(id) as ActivityRow[]); +} + +export function latestFailureEvent(id: string): ProvisioningEvent | null { + const row = db().prepare(`SELECT id, created_at, message, kind, phase_key, details_json + FROM server_deployment_events WHERE deployment_id = ? + AND (kind IN ('failed', 'interrupted') OR (kind = 'outcome' + AND CASE WHEN json_valid(details_json) THEN json_extract(details_json, '$.state') END IN ('failed', 'interrupted')) + OR (kind IS NULL AND (message LIKE 'Failed:%' OR message LIKE 'Interrupted by application restart.%'))) + ORDER BY id DESC LIMIT 1`).get(id) as ActivityRow | undefined; + return row ?? null; +} diff --git a/src/lib/server/provisioning/history.ts b/src/lib/server/provisioning/history.ts index 731f4d8..dfd166c 100644 --- a/src/lib/server/provisioning/history.ts +++ b/src/lib/server/provisioning/history.ts @@ -2,20 +2,35 @@ import { randomUUID } from 'node:crypto'; import { db } from '../db/index.js'; import { recipeForDeployment, summarizeRecipe } from '../fleet/launch-recipes.js'; import { actionLabels, type ProvisioningAction } from '../../provisioning.js'; +import type { ProvisioningCapacity, ProvisioningDeployment, ProvisioningDetail, ProvisioningSource } from '../../provisioning-detail.js'; +import { fleetSettings } from '../settings.js'; +import { activitySummary, deploymentEvents, latestFailureEvent } from './activity.js'; +import { readProgress, recordProgress, type ProgressUpdate } from './progress.js'; export function beginDeployment(server: string, action: ProvisioningAction, triggeredBy = 'ui'): string { const id = randomUUID(); - db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by) - VALUES (?, ?, ?, 'running', 'Starting', ?, ?)`).run(id, server, action, new Date().toISOString(), triggeredBy); - recordDeploymentEvent(id, action === 'create' ? 'Server creation requested' - : action === 'destroy' ? 'Server destruction requested' : `${actionLabels[action]} server requested`); + db().transaction(() => { + db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by) + VALUES (?, ?, ?, 'running', 'Starting', ?, ?)`).run(id, server, action, new Date().toISOString(), triggeredBy); + recordProgress(id, { phase: 'requested', observedAt: null, checkpoint: { key: 'requested', phase: 'requested', + label: action === 'create' ? 'Server creation requested' + : action === 'destroy' ? 'Server destruction requested' : `${actionLabels[action]} server requested` } }); + })(); return id; } -export function deploymentStage(id: string, stage: string): void { - const previous = db().prepare('SELECT stage FROM server_deployments WHERE id = ?').get(id) as { stage: string } | undefined; - db().prepare('UPDATE server_deployments SET stage = ? WHERE id = ?').run(stage, id); - if (previous && previous.stage !== stage) recordDeploymentEvent(id, stage); +export function deploymentStage(id: string, stage: string, update?: ProgressUpdate): void { + db().transaction(() => { + const previous = db().prepare('SELECT stage FROM server_deployments WHERE id = ?').get(id) as { stage: string } | undefined; + const changed = !!previous && previous.stage !== stage; + const before = update && changed ? activitySummary(id).latestId ?? 0 : 0; + db().prepare('UPDATE server_deployments SET stage = ? WHERE id = ?').run(stage, id); + if (update) recordProgress(id, update); + // Structured transitions may already carry the exact legacy stage text. + const alreadyRecorded = update && changed && db().prepare(`SELECT 1 FROM server_deployment_events + WHERE deployment_id = ? AND id > ? AND message = ? LIMIT 1`).get(id, before, stage); + if (changed && !alreadyRecorded) recordDeploymentEvent(id, stage); + })(); } export function recordDeploymentEvent(id: string, message: string): void { @@ -23,19 +38,83 @@ export function recordDeploymentEvent(id: string, message: string): void { .run(id, new Date().toISOString(), message); } -export function getDeployment(id: string) { - const deployment = db().prepare(`SELECT d.*, j.expected_map, (d.server_deleted_at IS NULL AND EXISTS(SELECT 1 FROM servers s WHERE s.name = d.server AND s.destroyed_at IS NULL)) AS server_exists - FROM server_deployments d LEFT JOIN server_creation_jobs j ON j.deployment_id = d.id WHERE d.id = ?`).get(id); - if (!deployment) return null; - const events = db().prepare('SELECT id, created_at, message FROM server_deployment_events WHERE deployment_id = ? ORDER BY id').all(id); - const sources = db().prepare('SELECT kind, source_id, server FROM provisioning_sources WHERE operation_id = ? ORDER BY kind, source_id').all(id); - const launch = recipeForDeployment(id); - const collection = db().prepare(`SELECT c.id, c.name FROM ( - SELECT collection_id, deployment_id FROM server_collection_members - UNION ALL SELECT collection_id, deployment_id FROM server_collection_removals) m - JOIN server_launch_collections c ON c.id = m.collection_id WHERE m.deployment_id = ?`).get(id) ?? null; - return { deployment, events, sources, recipe: launch ? summarizeRecipe(launch.recipe) : null, blueprintName: launch?.blueprintName ?? null, - canResumeVerification: canResumeVerification(id), canResumeCleanup: canResumeCleanup(id), canResumeRebuild: canResumeRebuild(id), canCancelRemoval: canCancelRemoval(id), collection }; +function warningMessages(raw: string): string[] { + try { + const parsed: unknown = JSON.parse(raw); + if (!Array.isArray(parsed)) return ['Recorded warnings could not be read.']; + const messages = parsed.filter((value): value is string => typeof value === 'string'); + if (messages.length !== parsed.length) messages.push('Some recorded warnings could not be read.'); + return messages; + } catch { return ['Recorded warnings could not be read.']; } +} + +function sourceDetails(source: Omit): ProvisioningSource { + const id = source.source_id; + if (source.kind === 'campaign') { + const row = db().prepare('SELECT action FROM deployments WHERE id = ?').get(id) as { action: string } | undefined; + const action = row?.action === 'promote' ? 'Promotion' : row?.action === 'rollback' ? 'Rollback' : row?.action === 'drift' ? 'Reconciliation' : 'Campaign'; + return { ...source, label: `${action} ${id.slice(0, 8)}`, href: row ? `/campaigns/${encodeURIComponent(id)}` : null }; + } + if (source.kind === 'beta') { + const row = db().prepare('SELECT notes FROM betas WHERE id = ?').get(id) as { notes: string | null } | undefined; + const label = row?.notes?.trim().split('\n')[0].slice(0, 120); + return { ...source, label: label ? `Beta #${id}: ${label}` : `Beta #${id}`, href: row ? `/betas/${encodeURIComponent(id)}` : null }; + } + if (source.kind === 'watchdog') { + const row = db().prepare('SELECT summary FROM watchdog_incidents WHERE id = ?').get(id) as { summary: string } | undefined; + return { ...source, label: row?.summary ?? `Maintenance finding ${id.slice(0, 8)}`, href: row ? `/maintenance/${encodeURIComponent(id)}` : null }; + } + const row = db().prepare('SELECT kind FROM maintenance_jobs WHERE id = ?').get(id) as { kind: string } | undefined; + return { ...source, label: row && row.kind !== 'legacy' ? `Maintenance ${row.kind} #${id}` : `Maintenance #${id}`, + href: row ? `/maintenance?view=activity#job-${encodeURIComponent(id)}` : null }; +} + +/** These are current pool snapshots, not a second admission/eligibility engine. */ +function capacityFor(deployment: ProvisioningDeployment, evaluatedAt: string): ProvisioningCapacity { + const settings = fleetSettings(); + const pools: ProvisioningCapacity['pools'] = []; + if (deployment.action === 'create' || deployment.action === 'rebuild') { + const jobs = db().prepare(`SELECT id, server, state FROM server_deployments + WHERE server_deleted_at IS NULL AND action IN ('create','rebuild') AND state IN ('queued','running','awaiting-health') + ORDER BY started_at, rowid`).all() as { id: string; server: string; state: string }[]; + const holders = jobs.filter(job => job.state !== 'queued').map(({ id, server }) => ({ id, server })); + const order = jobs.filter(job => job.state === 'queued').findIndex(job => job.id === deployment.id); + pools.push({ kind: 'startup', limit: settings.createConcurrency, used: holders.length, holders, + requestOrder: order < 0 ? null : order + 1, ownsSlot: holders.some(holder => holder.id === deployment.id) }); + } + if (['start', 'restart', 'rebuild'].includes(deployment.action)) { + const holders = db().prepare(`SELECT operation_id AS id, server FROM restart_rollout WHERE healthy_at IS NULL + ORDER BY requested_at, server`).all() as { id: string | null; server: string }[]; + pools.push({ kind: 'restart', limit: settings.restartConcurrency, used: holders.length, holders, + requestOrder: null, ownsSlot: holders.some(holder => holder.id === deployment.id) }); + } + return { evaluatedAt, pools }; +} + +/** The default retains the full legacy event response. Page clients opt into a bounded window. */ +export function getDeployment(id: string, options: { events?: 'all' | 'recent' } = {}): ProvisioningDetail | null { + return db().transaction(() => { + const row = db().prepare(`SELECT d.*, j.expected_map, (d.server_deleted_at IS NULL AND EXISTS(SELECT 1 FROM servers s WHERE s.name = d.server AND s.destroyed_at IS NULL)) AS server_exists + FROM server_deployments d LEFT JOIN server_creation_jobs j ON j.deployment_id = d.id WHERE d.id = ?`).get(id) as + (ProvisioningDeployment & { observation_json: string | null; phase_key: string | null; phase_started_at: string | null; observed_at: string | null }) | undefined; + if (!row) return null; + const generatedAt = new Date().toISOString(); + const { observation_json: _snapshot, phase_key: _phase, phase_started_at: _phaseStarted, observed_at: _observed, ...deployment } = row; + const sources = (db().prepare('SELECT kind, source_id, server FROM provisioning_sources WHERE operation_id = ? ORDER BY kind, source_id') + .all(id) as Omit[]).map(sourceDetails); + const launch = recipeForDeployment(id); + const collection = db().prepare(`SELECT c.id, c.name FROM ( + SELECT collection_id, deployment_id FROM server_collection_members + UNION ALL SELECT collection_id, deployment_id FROM server_collection_removals) m + JOIN server_launch_collections c ON c.id = m.collection_id WHERE m.deployment_id = ? ORDER BY c.created_at, c.id LIMIT 1`).get(id) as { id: string; name: string } | undefined; + return { deployment, events: deploymentEvents(id, options.events === 'recent'), + activity: { ...activitySummary(id), failureExcerpt: latestFailureEvent(id) }, sources, + recipe: launch ? summarizeRecipe(launch.recipe) : null, blueprintName: launch?.blueprintName ?? null, + canResumeVerification: canResumeVerification(id), canResumeCleanup: canResumeCleanup(id), + canResumeRebuild: canResumeRebuild(id), canCancelRemoval: canCancelRemoval(id), collection: collection ?? null, + progress: readProgress(row), capacity: capacityFor(deployment, generatedAt), + warningMessages: warningMessages(deployment.warnings), generatedAt }; + })(); } // This stage is written only after Komodo has confirmed deployment deletion. @@ -48,10 +127,13 @@ export function canCancelRemoval(id: string): boolean { /** Competes atomically with the worker's claim, before any external removal. */ export function cancelRemoval(id: string): boolean { return db().transaction(() => { + const finishedAt = new Date().toISOString(); const result = db().prepare(`UPDATE server_deployments SET state = 'cancelled', stage = 'Removal cancelled', finished_at = ?, error = NULL - WHERE id = ? AND server_deleted_at IS NULL AND action = 'destroy' AND state = 'waiting' AND stage != ?`).run(new Date().toISOString(), id, REMOVING_SERVER); + WHERE id = ? AND server_deleted_at IS NULL AND action = 'destroy' AND state = 'waiting' AND stage != ?`).run(finishedAt, id, REMOVING_SERVER); if (!result.changes) return false; - recordDeploymentEvent(id, 'Queued removal cancelled. The server and its files were kept.'); + recordProgress(id, { blocker: null, checks: [], event: { kind: 'outcome', + message: 'Queued removal cancelled. The server and its files were kept.', + details: { state: 'cancelled', finishedAt } } }); return true; })(); } @@ -76,8 +158,10 @@ export function canResumeCleanup(id: string): boolean { export function resumeCleanup(id: string): boolean { return db().transaction(() => { if (!canResumeCleanup(id)) return false; + preserveOutcome(id); db().prepare("UPDATE server_deployments SET state = 'waiting', finished_at = NULL, error = NULL WHERE id = ?").run(id); - recordDeploymentEvent(id, 'Resumed local cleanup after confirmed Komodo deletion'); + recordProgress(id, { phase: 'cleanup', blocker: null, checks: [], observedAt: null, + event: { kind: 'resumed', message: 'Resumed local cleanup after confirmed Komodo deletion' } }); return true; })(); } @@ -100,29 +184,55 @@ export function canResumeRebuild(id: string): boolean { } export function resumeRebuild(id: string): boolean { - if (!canResumeRebuild(id)) return false; - db().prepare("UPDATE server_deployments SET state = 'waiting', finished_at = NULL, error = NULL WHERE id = ?").run(id); - recordDeploymentEvent(id, 'Resumed rebuild cleanup after confirmed Komodo deletion'); - return true; + return db().transaction(() => { + if (!canResumeRebuild(id)) return false; + preserveOutcome(id); + db().prepare("UPDATE server_deployments SET state = 'waiting', finished_at = NULL, error = NULL WHERE id = ?").run(id); + recordProgress(id, { phase: 'cleanup', blocker: null, checks: [], observedAt: null, + event: { kind: 'resumed', message: 'Resumed rebuild cleanup after confirmed Komodo deletion' } }); + return true; + })(); } /** Resume observation only. Never repeat a container creation or restart. */ export function resumeVerification(id: string): boolean { return db().transaction(() => { if (!canResumeVerification(id)) return false; + preserveOutcome(id); db().prepare("UPDATE server_deployments SET state = 'awaiting-health', finished_at = NULL, error = NULL WHERE id = ?").run(id); - recordDeploymentEvent(id, 'Resumed verification of the existing server'); - deploymentStage(id, 'Waiting for Komodo health'); + deploymentStage(id, 'Waiting for Komodo health', { phase: 'verify-container', blocker: null, checks: [], observedAt: null, + event: { kind: 'resumed', message: 'Resumed verification of the existing server' } }); return true; })(); } export function finishDeployment(id: string, result: { ok: boolean; message?: string; warnings?: string[] }): void { const state = !result.ok ? 'failed' : result.warnings?.length ? 'warning' : 'done'; - db().prepare(`UPDATE server_deployments SET state = ?, finished_at = ?, error = ?, warnings = ? WHERE id = ?`) - .run(state, new Date().toISOString(), result.message ?? null, JSON.stringify(result.warnings ?? []), id); - for (const warning of result.warnings ?? []) recordDeploymentEvent(id, warning); - recordDeploymentEvent(id, result.ok ? (state === 'warning' ? 'Completed with warnings' : 'Completed') : `Failed: ${result.message ?? 'Unknown error'}`); + db().transaction(() => { + const finishedAt = new Date().toISOString(); + db().prepare(`UPDATE server_deployments SET state = ?, finished_at = ?, error = ?, warnings = ? WHERE id = ?`) + .run(state, finishedAt, result.message ?? null, JSON.stringify(result.warnings ?? []), id); + for (const warning of result.warnings ?? []) recordProgress(id, { event: { kind: 'warning', message: warning } }); + recordProgress(id, { blocker: null, event: { kind: 'outcome', + message: result.ok ? (state === 'warning' ? 'Completed with warnings' : 'Completed') : `Failed: ${result.message ?? 'Unknown error'}`, + details: { state, finishedAt, error: result.message ?? null, warnings: result.warnings ?? [] } } }); + })(); +} + +/** Legacy failures also need a durable structured result before same-ID resume clears it. */ +function preserveOutcome(id: string): void { + const row = db().prepare('SELECT state, finished_at, error, warnings, observation_json FROM server_deployments WHERE id = ?').get(id) as + { state: string; finished_at: string | null; error: string | null; warnings: string; observation_json: string | null } | undefined; + if (!row || !['failed', 'interrupted', 'warning', 'done', 'cancelled'].includes(row.state)) return; + const previous = db().prepare("SELECT details_json FROM server_deployment_events WHERE deployment_id = ? AND kind = 'outcome' ORDER BY id DESC LIMIT 1") + .get(id) as { details_json: string | null } | undefined; + try { + const details = JSON.parse(previous?.details_json ?? 'null'); + if (details?.state === row.state && details?.finishedAt === row.finished_at) return; + } catch { /* Preserve the row's known outcome when old event metadata is missing. */ } + recordProgress(id, { observedAt: readProgress(row)?.observedAt ?? null, event: { kind: 'outcome', + message: `${row.state === 'interrupted' ? 'Interrupted' : 'Recorded outcome'}: ${row.error ?? row.state}`, + details: { state: row.state, finishedAt: row.finished_at, error: row.error, warnings: warningMessages(row.warnings) } } }); } export function listDeployments() { @@ -132,9 +242,12 @@ export function listDeployments() { } export function interruptDeployments(): void { - const running = db().prepare("SELECT id FROM server_deployments WHERE state = 'running'").all() as { id: string }[]; - db().prepare(`UPDATE server_deployments SET state = 'interrupted', finished_at = ?, - error = 'The application restarted before this operation finished. Check the server before retrying.' - WHERE state = 'running'`).run(new Date().toISOString()); - for (const { id } of running) recordDeploymentEvent(id, 'Interrupted by application restart. Check the server before retrying.'); + db().transaction(() => { + const running = db().prepare("SELECT id FROM server_deployments WHERE state = 'running'").all() as { id: string }[]; + const finishedAt = new Date().toISOString(); + const error = 'The application restarted before this operation finished. Check the server before retrying.'; + db().prepare(`UPDATE server_deployments SET state = 'interrupted', finished_at = ?, error = ? WHERE state = 'running'`).run(finishedAt, error); + for (const { id } of running) recordProgress(id, { blocker: null, event: { kind: 'outcome', + message: 'Interrupted by application restart. Check the server before retrying.', details: { state: 'interrupted', finishedAt, error } } }); + })(); } diff --git a/src/lib/server/provisioning/index.ts b/src/lib/server/provisioning/index.ts index bf5d7c4..71d8932 100644 --- a/src/lib/server/provisioning/index.ts +++ b/src/lib/server/provisioning/index.ts @@ -4,4 +4,5 @@ export { requestLifecycle, requestRestart, requestWatchdogLifecycle, escalateWat export type { LifecycleOperation, RestartSource } from './requests.js'; export { queueCreateServer, destroyServer, previewCreation, creationQueue, nextAvailablePort, nextLobbyNumber } from './instances.js'; export { getDeployment, listDeployments, resumeVerification, resumeCleanup, resumeRebuild, cancelRemoval } from './history.js'; +export { deploymentActivity } from './activity.js'; export { restartSettling, restartRecoveryPending } from './restarts.js'; diff --git a/src/lib/server/provisioning/instances.ts b/src/lib/server/provisioning/instances.ts index 2bcd389..06da182 100644 --- a/src/lib/server/provisioning/instances.ts +++ b/src/lib/server/provisioning/instances.ts @@ -1,5 +1,5 @@ import { fleetSettings } from '../settings.js'; -import { probePlayerCount, probeServerStatus } from '../fleet/status-evidence.js'; +import { probePlayerEvidence, probeServerStatus } from '../fleet/status-evidence.js'; import { beginDeployment, deploymentStage, finishDeployment, recordDeploymentEvent, REMOVING_SERVER, resumeCleanup, resumeVerification, claimRemoval } from './history.js'; import { randomUUID } from 'node:crypto'; import type { LaunchPreview } from '$lib/deployment-plan.js'; @@ -25,6 +25,8 @@ import { import { MAP_CONFIG_FILE } from '../fleet/server-config.js'; import { recordScript } from '../fleet/inventory.js'; import { getVersion } from '../fleet/scripts.js'; +import type { ProvisioningBlocker } from '../../provisioning-detail.js'; +import { recordProgress, type ProgressUpdate } from './progress.js'; export type CreateServerFailure = | 'bad-request' @@ -147,6 +149,13 @@ function buildSpec(args: { } let creating = false; +async function observeCommand(operation: string, command: () => Promise): Promise { + try { return await command(); } + catch (err) { + recordProgress(operation, { command: { state: 'uncertain', message: 'No confirmed command result was received: ' + (err as Error).message } }); + throw err; + } +} interface PreparedCreation { ok: true; name: string; port: number; plutoKey: string; mapEntry: ZombieMapEntry; } @@ -225,7 +234,8 @@ export function queueCreateServer(input: CreateServerInput, releasingServers: st db().prepare("UPDATE server_deployments SET state = 'queued' WHERE id = ?").run(id); db().prepare('INSERT INTO server_creation_jobs (deployment_id, server, port, game_mode, map_slug, expected_map) VALUES (?, ?, ?, ?, ?, ?)') .run(id, prepared.name, prepared.port, input.gameMode, prepared.mapEntry.slug, prepared.mapEntry.canonical); - deploymentStage(id, 'Queued for server creation'); + deploymentStage(id, 'Queued for server creation', { phase: 'startup-queue', recipeStatus: 'queued', observedAt: null, + blocker: { code: 'startup-queue', message: 'Waiting for startup admission and available capacity.', automatic: true } }); return { ok: true, name: prepared.name, port: prepared.port, deploymentId: id }; })(); } @@ -252,7 +262,8 @@ export function queueRebuildServer(name: string, triggeredBy = 'ui'): CreateServ .run(id, name, server.game_port, 'zombie', map.slug, map.canonical); db().prepare('INSERT INTO server_rebuild_jobs (deployment_id) VALUES (?)').run(id); db().prepare('INSERT INTO server_launch_recipes (deployment_id, recipe) VALUES (?, ?)').run(id, JSON.stringify(recipe)); - deploymentStage(id, 'Waiting to rebuild the empty server'); + deploymentStage(id, 'Waiting to rebuild the empty server', { phase: 'eligibility', recipeStatus: 'queued', observedAt: null, + blocker: { code: 'eligibility', message: 'Waiting for safe rebuild eligibility checks.', automatic: true } }); return { ok: true as const, name, port: server.game_port!, deploymentId: id }; })(); } @@ -266,12 +277,15 @@ export async function rebuildServer(id: string): Promise { if (blocker) { // Keep the cleanup checkpoint intact when resuming after remote deletion. if (job.stage === REMOVING_SERVER) finishDeployment(id, { ok: false, message: blocker }); - else deploymentStage(id, blocker); + else deploymentStage(id, blocker, { phase: 'eligibility', blocker: betaRemovalEvidence(job.server, blocker), checks: [] }); return; } if (creating || destroying.size || creationQueue().active.length >= fleetSettings().createConcurrency) { // The cleanup checkpoint must survive capacity waits and app restarts. - if (job.stage !== REMOVING_SERVER) deploymentStage(id, 'Waiting for a server creation slot before rebuilding'); + const progress: ProgressUpdate = { phase: job.stage === REMOVING_SERVER ? 'cleanup' : 'eligibility', + blocker: { code: 'startup-capacity', message: 'Waiting for a server creation slot before rebuilding', automatic: true } }; + if (job.stage !== REMOVING_SERVER) deploymentStage(id, 'Waiting for a server creation slot before rebuilding', progress); + else recordProgress(id, progress); return; } // Recheck prerequisites while the original instance still exists. @@ -290,7 +304,9 @@ export async function rebuildServer(id: string): Promise { db().transaction(() => { db().prepare('UPDATE server_rebuild_jobs SET removed_at = ? WHERE deployment_id = ?').run(new Date().toISOString(), id); db().prepare("UPDATE server_deployments SET state = 'queued' WHERE id = ?").run(id); - deploymentStage(id, 'Queued for server creation'); + deploymentStage(id, 'Queued for server creation', { phase: 'startup-queue', checks: [], + blocker: { code: 'startup-queue', message: 'Old instance removed; waiting for fresh-instance startup admission.', automatic: true }, + checkpoint: { key: 'cleanup', label: 'Old instance files and registry removed' } }); })(); } catch (err) { finishDeployment(id, { ok: false, message: (err as Error).message }); } finally { destroying.delete(job.server); } @@ -369,11 +385,15 @@ async function createServerUnlocked(input: CreateServerInput, prepared: Prepared // Resolve once more at provisioning so promotions made while queued also apply. // Persist the exact attempted files for history, never as input to future launches. const recipe = captureRecipe(game, mapEntry.slug); - db().prepare(`INSERT INTO server_launch_recipes (deployment_id, recipe) VALUES (?, ?) - ON CONFLICT(deployment_id) DO UPDATE SET recipe = excluded.recipe`) - .run(operation, JSON.stringify(recipe)); + db().transaction(() => { + db().prepare(`INSERT INTO server_launch_recipes (deployment_id, recipe) VALUES (?, ?) + ON CONFLICT(deployment_id) DO UPDATE SET recipe = excluded.recipe`) + .run(operation, JSON.stringify(recipe)); + deploymentStage(operation, 'Preparing server files', { phase: 'prepare', recipeStatus: 'attempted', blocker: null, checks: [], + checkpoint: { key: 'startup-queue', label: 'Admitted to startup' }, + event: { kind: 'recipe', message: 'Promoted files selected for this creation attempt.', details: { capturedAt: recipe.capturedAt } } }); + })(); const configFile = recipe.configFile; - deploymentStage(operation, 'Preparing server files'); const hostDir = serverDir(name); // Step 1: filesystem. mkdir mode is masked by umask, so chmod explicitly @@ -398,13 +418,18 @@ async function createServerUnlocked(input: CreateServerInput, prepared: Prepared } // Step 2: Komodo CreateDeployment. - deploymentStage(operation, 'Creating Komodo deployment'); + deploymentStage(operation, 'Creating Komodo deployment', { phase: 'create-deployment', + checkpoint: { key: 'prepare', label: 'Server files prepared' }, + command: { state: 'attempted', message: 'Komodo deployment creation attempted.' } }); const spec = buildSpec({ game, name, port, hostDir, plutoKey, recipe }); - const createRes = await komodoCreateDeployment(spec); + const createRes = await observeCommand(operation, () => komodoCreateDeployment(spec)); if (!createRes.ok) { + recordProgress(operation, { command: { state: createRes.kind === 'unreachable' || (createRes.httpStatus !== undefined && createRes.httpStatus >= 500) ? 'uncertain' : 'rejected', message: createRes.message } }); await rm(hostDir, { recursive: true, force: true }).catch(() => {}); return { ok: false, kind: 'komodo', message: `CreateDeployment: ${createRes.message}` }; } + recordProgress(operation, { command: { state: 'accepted', message: 'Komodo created the deployment.' }, + checkpoint: { key: 'remote-deployment-created', phase: 'create-deployment', label: 'Komodo deployment created' } }); // Step 3: DB row — insert *before* execute/Deploy so the sweep sees the // server immediately even if Deploy hangs. @@ -446,23 +471,31 @@ async function createServerUnlocked(input: CreateServerInput, prepared: Prepared void applyDeploymentTags(name, game, mapEntry.canonical); // Step 5: execute/Deploy — actually start the container. - deploymentStage(operation, 'Starting container'); - db().prepare('UPDATE server_deployments SET submitted_at = ? WHERE id = ?').run(new Date().toISOString(), operation); - const deployRes = await komodoDeploy(name); + db().transaction(() => { + deploymentStage(operation, 'Starting container', { phase: 'container-action', + checkpoint: { key: 'create-deployment', label: 'Deployment created and server registered' }, + command: { state: 'attempted', message: 'Container startup submission attempted.' } }); + db().prepare('UPDATE server_deployments SET submitted_at = ? WHERE id = ?').run(new Date().toISOString(), operation); + })(); + const deployRes = await observeCommand(operation, () => komodoDeploy(name)); if (!deployRes.ok) { if (deployRes.kind === 'unreachable' || (deployRes.httpStatus !== undefined && deployRes.httpStatus >= 500)) { // A timeout or server error may arrive after the remote boot began. // Preserve files, identity, and capacity until readiness is known. recordDeploymentEvent(operation, 'Startup response uncertain; verifying the existing server: ' + deployRes.message); + recordProgress(operation, { command: { state: 'uncertain', message: 'Startup may have begun: ' + deployRes.message } }); return { ok: true, name, port, deploymentId: operation }; } // Container never started — undo everything so retrying is clean. + recordProgress(operation, { command: { state: 'rejected', message: 'Startup request rejected: ' + deployRes.message } }); await komodoDeleteDeployment(name).catch(() => {}); db().prepare('DELETE FROM servers WHERE name = ?').run(name); await rm(hostDir, { recursive: true, force: true }).catch(() => {}); return { ok: false, kind: 'komodo', message: `Deploy: ${deployRes.message}` }; } + recordProgress(operation, { command: { state: 'accepted', message: 'Komodo accepted startup; readiness still requires verification.' }, + checkpoint: { key: 'container-action', label: 'Container startup accepted' } }); return { ok: true, name, port, deploymentId: operation }; } @@ -515,7 +548,8 @@ export function destroyServer(name: string, triggeredBy = 'ui'): DestroyServerRe const operation = db().transaction(() => { const id = beginDeployment(name, 'destroy', triggeredBy); db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id); - deploymentStage(id, 'Waiting to verify the server is empty'); + deploymentStage(id, 'Waiting to verify the server is empty', { phase: 'eligibility', observedAt: null, + blocker: { code: 'eligibility', message: 'Waiting for safe removal eligibility checks.', automatic: true } }); return id; })(); return { ok: true, warnings: [], deploymentId: operation }; @@ -528,6 +562,13 @@ export function processDestructions(): Promise { return destructionTick; } +function betaRemovalEvidence(server: string, message: string): ProvisioningBlocker { + const beta = db().prepare(`SELECT b.id FROM beta_servers bs JOIN betas b ON b.id = bs.beta_id + WHERE bs.server = ? AND bs.server_deleted_at IS NULL AND b.state IN ('running','ready-promote') ORDER BY b.id LIMIT 1`) + .get(server) as { id: number } | undefined; + return { code: 'active-beta', message, automatic: true, href: beta ? `/betas/${beta.id}` : null, label: 'Blocking beta' }; +} + async function runDestructions(): Promise { const queued = db().prepare(`SELECT id, server, stage, server_deleted_at FROM server_deployments WHERE action = 'destroy' AND state = 'waiting' AND (server_deleted_at IS NULL OR stage = ?) ORDER BY started_at`).all(REMOVING_SERVER) as { id: string; server: string; stage: string; server_deleted_at: string | null }[]; @@ -553,37 +594,53 @@ async function runDestructions(): Promise { } if (!getServer(operation.server)) { if (hasActiveCreation(operation.server)) { - deploymentStage(operation.id, 'Waiting for server creation to finish'); + const creatingOperation = db().prepare(`SELECT id FROM server_deployments WHERE server = ? AND server_deleted_at IS NULL + AND action IN ('create','rebuild') AND state IN ('queued','running','awaiting-health') LIMIT 1`).get(operation.server) as { id: string } | undefined; + deploymentStage(operation.id, 'Waiting for server creation to finish', { phase: 'eligibility', checks: [], + blocker: { code: 'creation-pending', message: 'Waiting for server creation to finish', automatic: true, + href: creatingOperation ? `/provisioning/${encodeURIComponent(creatingOperation.id)}` : null, label: 'Creation operation' } }); continue; } finishDeployment(operation.id, { ok: false, message: 'Server is no longer registered' }); continue; } const blocker = activeBetaRemovalBlocker(operation.server); - if (blocker) { deploymentStage(operation.id, blocker); continue; } + if (blocker) { deploymentStage(operation.id, blocker, { phase: 'eligibility', checks: [], blocker: betaRemovalEvidence(operation.server, blocker) }); continue; } // A fresh successful zero is required; cached occupancy and timeouts are not evidence of emptiness. - let players = await probePlayerCount(operation.server); + const evidence = await probePlayerEvidence(operation.server); + let players = evidence.players; + let occupancy: NonNullable[number] = { key: 'occupancy', label: 'Player occupancy', + result: players === 0 ? 'pass' : players === null ? 'unknown' : 'waiting', source: evidence.source, + checkedAt: evidence.observedAt, receivedAt: evidence.receivedAt, expected: 0, actual: players }; if (players === null) { const health = await komodoInspectHealthy(operation.server); - if (['stopped','exited','created','dead'].includes(health.state?.toLowerCase() ?? '')) players = 0; + if (['stopped','exited','created','dead'].includes(health.state?.toLowerCase() ?? '')) { + players = 0; + occupancy = { key: 'stopped', label: 'Stopped container permits removal', result: 'pass', source: 'komodo', actual: health.state ?? null }; + } } if (!db().prepare("SELECT id FROM server_deployments WHERE id = ? AND state = 'waiting'").get(operation.id)) continue; // Beta ownership may have changed while the network probes were in flight. const latestBlocker = activeBetaRemovalBlocker(operation.server); - if (latestBlocker) { deploymentStage(operation.id, latestBlocker); continue; } + if (latestBlocker) { deploymentStage(operation.id, latestBlocker, { phase: 'eligibility', checks: [occupancy], blocker: betaRemovalEvidence(operation.server, latestBlocker) }); continue; } if (players !== 0) { - deploymentStage(operation.id, players === null + const message = players === null ? 'Waiting for player count: server could not be reached' - : 'Waiting for server to empty: ' + players + ' player(s) online'); + : 'Waiting for server to empty: ' + players + ' player(s) online'; + deploymentStage(operation.id, message, { phase: 'eligibility', checks: [occupancy], + blocker: { code: players === null ? 'players-unknown' : 'players-present', message, automatic: true } }); continue; } if (!claimRemoval(operation.id)) continue; - deploymentStage(operation.id, 'Server is empty (0 players verified)'); + deploymentStage(operation.id, 'Server is empty (0 players verified)', { phase: 'eligibility', checks: [occupancy], blocker: null, + checkpoint: { key: 'eligibility', label: 'Removal eligibility verified' } }); const result = await destroyServerUnlocked(operation.server, operation.id); finishDeployment(operation.id, result); } catch (err) { const row = db().prepare('SELECT state FROM server_deployments WHERE id = ?').get(operation.id) as { state: string }; - if (row.state === 'waiting') deploymentStage(operation.id, 'Waiting for player count: ' + (err as Error).message); + if (row.state === 'waiting') deploymentStage(operation.id, 'Waiting for player count: ' + (err as Error).message, { + phase: 'eligibility', checks: [], blocker: { code: 'players-unknown', message: (err as Error).message, automatic: true } + }); else if (row.state !== 'cancelled') finishDeployment(operation.id, { ok: false, message: (err as Error).message }); } finally { destroying.delete(operation.server); } } @@ -601,11 +658,15 @@ async function destroyServerUnlocked(name: string, operation: string): Promise komodoDestroyDeployment(name)); if (!destroyRes.ok) { + recordProgress(operation, { command: { state: destroyRes.kind === 'unreachable' || (destroyRes.httpStatus !== undefined && destroyRes.httpStatus >= 500) ? 'uncertain' : 'rejected', message: destroyRes.message } }); return { ok: false, kind: 'komodo', message: 'DestroyDeployment failed: ' + destroyRes.message, warnings }; } + recordProgress(operation, { command: { state: 'accepted', message: 'Komodo accepted teardown; removal is not yet confirmed.' } }); // 2: wait for the container to actually disappear. Komodo refuses to // DeleteDeployment while a container still exists for it, so poll the @@ -619,12 +680,19 @@ async function destroyServerUnlocked(name: string, operation: string): Promise komodoDeleteDeployment(name)); if (!deleteRes.ok) { + recordProgress(operation, { command: { state: deleteRes.kind === 'unreachable' || (deleteRes.httpStatus !== undefined && deleteRes.httpStatus >= 500) ? 'uncertain' : 'rejected', message: deleteRes.message } }); return { ok: false, kind: 'komodo', message: 'DeleteDeployment failed: ' + deleteRes.message, warnings }; } + deploymentStage(operation, REMOVING_SERVER, { phase: 'cleanup', blocker: null, checks: [], + command: { state: 'accepted', message: 'Komodo confirmed deployment deletion.' }, + checkpoints: [{ key: 'remove-container', label: 'Container removal confirmed by deployment deletion' }, + { key: 'delete-deployment', label: 'Komodo deployment deletion confirmed' }] }); + return cleanupDestroyedServer(name, operation); } @@ -635,12 +703,14 @@ async function cleanupDestroyedServer(name: string, operation: string): Promise< const warnings: string[] = []; // Persist the checkpoint before local work so failed cleanup can resume without RCON. // Legacy operations already recorded this same stage after Komodo deletion. - deploymentStage(operation, REMOVING_SERVER); + deploymentStage(operation, REMOVING_SERVER, { phase: 'cleanup', blocker: null, checks: [], + checkpoint: { key: 'delete-deployment', label: 'Komodo deployment deletion confirmed' } }); // Host directory. Fall back to derived path if host_dir wasn't set on // legacy rows. const hostDir = row?.host_dir ?? serverDir(name); try { await rm(hostDir, { recursive: true, force: true }); + recordProgress(operation, { checkpoint: { key: 'files-removed', label: 'Instance files removed', phase: 'cleanup' } }); } catch (err) { warnings.push(`rm ${hostDir} failed: ${(err as Error).message}`); } @@ -648,7 +718,14 @@ async function cleanupDestroyedServer(name: string, operation: string): Promise< // The registry trigger preserves history and marks its old server identity. // Live inventory, rollback snapshots, and restart slots still cascade away. try { - db().prepare('DELETE FROM servers WHERE name = ?').run(name); + db().transaction(() => { + const recovery = db().prepare('SELECT operation_id FROM restart_rollout WHERE server = ? AND healthy_at IS NULL').get(name) as { operation_id: string | null } | undefined; + db().prepare('DELETE FROM servers WHERE name = ?').run(name); + if (recovery?.operation_id) recordProgress(recovery.operation_id, { recovery: { state: 'released', operationId: recovery.operation_id, + message: 'Recovery capacity released when the old instance registry was removed.' } }); + recordProgress(operation, { checkpoint: { key: 'registry-removed', label: 'Server registration removed', phase: 'cleanup' } }); + if (!warnings.length) recordProgress(operation, { checkpoint: { key: 'cleanup', label: 'Local files and registry cleanup completed' } }); + })(); } catch (err) { return { ok: false, @@ -684,8 +761,11 @@ async function provisionCreation(id: string): Promise { { ok: true, name: job.server, port: job.port, mapEntry, plutoKey }, id); if (!result.ok) finishDeployment(id, result); else { - db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ?").run(id); - deploymentStage(id, 'Waiting for Komodo health'); + db().transaction(() => { + db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ?").run(id); + deploymentStage(id, 'Waiting for Komodo health', { phase: 'verify-container', checks: [], + blocker: { code: 'container-readiness', message: 'Waiting for container readiness.', automatic: true } }); + })(); } return result; } catch (err) { @@ -693,9 +773,13 @@ async function provisionCreation(id: string): Promise { if (server?.game_port === job.port) { // A lost response may have happened after Komodo started the server. // Keep its slot and observe readiness instead of starting another boot. - db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ?").run(id); - recordDeploymentEvent(id, 'Startup response interrupted: ' + (err as Error).message); - deploymentStage(id, 'Verifying existing server after interrupted startup'); + db().transaction(() => { + db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ?").run(id); + recordDeploymentEvent(id, 'Startup response interrupted: ' + (err as Error).message); + deploymentStage(id, 'Verifying existing server after interrupted startup', { phase: 'verify-container', checks: [], + command: { state: 'uncertain', message: 'Startup response interrupted: ' + (err as Error).message }, + blocker: { code: 'container-readiness', message: 'Observing the existing instance after an uncertain startup.', automatic: true } }); + })(); } else finishDeployment(id, { ok: false, message: (err as Error).message }); throw err; } finally { creating = false; } @@ -722,19 +806,32 @@ export function creationQueue() { return { active: jobs.filter(job => job.state !== 'queued'), queued: jobs.filter(job => job.state === 'queued') }; } -function creationBlocker(id: string): string | null { - const dependency = db().prepare(`SELECT d.server, d.state FROM server_creation_dependencies dep +function creationBlocker(id: string): ProvisioningBlocker | null { + const dependency = db().prepare(`SELECT d.id, d.server, d.state FROM server_creation_dependencies dep JOIN server_deployments d ON d.id = dep.removal_id - WHERE dep.deployment_id = ? AND d.state NOT IN ('done', 'warning') LIMIT 1`).get(id) as { server: string; state: string } | undefined; - if (dependency) return `Waiting for removal of ${dependency.server} (${dependency.state}) before reusing its port`; + WHERE dep.deployment_id = ? AND d.state NOT IN ('done', 'warning') LIMIT 1`).get(id) as { id: string; server: string; state: string } | undefined; + if (dependency) return { code: 'removal-dependency', message: `Waiting for removal of ${dependency.server} (${dependency.state}) before reusing its port`, + automatic: true, href: `/provisioning/${encodeURIComponent(dependency.id)}`, label: 'Required removal' }; const occupied = db().prepare(`SELECT s.name FROM servers s JOIN server_creation_jobs j ON s.game_port = j.port WHERE j.deployment_id = ? LIMIT 1`).get(id) as { name: string } | undefined; - return occupied ? `Waiting for port to be released by ${occupied.name}` : null; + return occupied ? { code: 'port-occupied', message: `Waiting for port to be released by ${occupied.name}`, automatic: true, + href: `/servers/${encodeURIComponent(occupied.name)}`, label: 'Server using this port' } : null; +} + +function stillCheckingCreation(id: string): boolean { + return !!db().prepare(`SELECT 1 FROM server_deployments d JOIN server_creation_jobs j ON j.deployment_id = d.id + JOIN servers s ON s.name = j.server AND s.game_port = j.port + WHERE d.id = ? AND d.server_deleted_at IS NULL AND d.state = 'awaiting-health'`).get(id); } async function runCreations(): Promise { const outcomes: CreationOutcomes = new Map(); - if (creating || destroying.size > 0) return outcomes; + if (creating || destroying.size > 0) { + for (const job of creationQueue().queued) recordProgress(job.id, { phase: 'startup-queue', blocker: { + code: 'startup-worker-busy', message: 'Waiting for in-flight instance creation or removal to finish.', automatic: true + } }); + return outcomes; + } // Restart recovery is observation-only: never repeat external creation or // deployment. A known interrupted startup must retain its readiness gate. const interrupted = db().prepare("SELECT id FROM server_deployments WHERE server_deleted_at IS NULL AND action IN ('create', 'rebuild') AND state = 'interrupted' ORDER BY started_at, rowid").all() as { id: string }[]; @@ -752,7 +849,8 @@ async function runCreations(): Promise { AND newer.rowid > d.rowid AND newer.action = 'create')`).get(id); if (removing) { db().prepare("UPDATE server_deployments SET state = 'awaiting-health', finished_at = NULL, error = NULL WHERE id = ?").run(id); - deploymentStage(id, 'Waiting for requested server removal after interrupted startup'); + deploymentStage(id, 'Waiting for requested server removal after interrupted startup', { checks: [], + blocker: { code: 'removal-pending', message: 'Waiting for the requested removal; the interrupted startup still occupies capacity.', automatic: true } }); } } @@ -766,21 +864,51 @@ async function runCreations(): Promise { finishDeployment(id, { ok: false, message: 'Server was removed or replaced before creation health checks completed' }); continue; } - if (db().prepare("SELECT id FROM server_deployments WHERE server = ? AND server_deleted_at IS NULL AND action = 'destroy' AND state IN ('waiting', 'running')").get(job.server)) continue; + const removal = db().prepare("SELECT id FROM server_deployments WHERE server = ? AND server_deleted_at IS NULL AND action = 'destroy' AND state IN ('waiting', 'running')").get(job.server) as { id: string } | undefined; + if (removal) { + recordProgress(id, { checks: [], blocker: { code: 'removal-pending', message: 'Creation verification is held while server removal is pending.', automatic: true, + href: `/provisioning/${encodeURIComponent(removal.id)}`, label: 'Removal operation' } }); + continue; + } + const checks: NonNullable = []; try { const komodo = await komodoInspectHealthy(job.server); - if (!komodo.ok) { deploymentStage(id, 'Waiting for Komodo health: ' + komodo.reason); continue; } + if (!stillCheckingCreation(id)) continue; + checks.push({ key: 'container', label: 'Container readiness', source: 'komodo', result: komodo.ok ? 'pass' : 'waiting', + checkedAt: new Date().toISOString(), actual: komodo.state ?? null, message: komodo.ok ? null : komodo.reason }); + if (!komodo.ok) { deploymentStage(id, 'Waiting for Komodo health: ' + komodo.reason, { phase: 'verify-container', checks, + blocker: { code: 'container-readiness', message: komodo.reason, automatic: true } }); continue; } const status = await probeServerStatus(job.server, job.submitted_at); - if (!status.ok) { deploymentStage(id, 'Waiting for RCON or fresh telemetry health: ' + status.reason); continue; } + if (!stillCheckingCreation(id)) continue; + checks.push({ key: 'game', label: 'Game status', source: status.ok ? status.source : null, result: status.ok ? 'pass' : 'waiting', + checkedAt: status.observedAt, receivedAt: status.ok ? status.receivedAt : null, actual: status.ok ? status.players : null, message: status.ok ? null : status.reason }); + if (!status.ok) { deploymentStage(id, 'Waiting for RCON or fresh telemetry health: ' + status.reason, { phase: 'verify-game', checks, + blocker: { code: 'game-readiness', message: status.reason, automatic: true } }); continue; } const actualMap = status.map; + checks.push({ key: 'map', label: 'Selected engine map', source: status.source, result: actualMap === job.expected_map ? 'pass' : 'waiting', + checkedAt: status.observedAt, receivedAt: status.receivedAt, expected: job.expected_map, actual: actualMap }); if (actualMap !== job.expected_map) { - deploymentStage(id, 'Waiting for selected map: expected ' + job.expected_map + ', ' + (status.source === 'rcon' ? 'RCON' : 'telemetry') + ' reports ' + (actualMap ?? 'no map')); + const message = 'Waiting for selected map: expected ' + job.expected_map + ', ' + (status.source === 'rcon' ? 'RCON' : 'telemetry') + ' reports ' + (actualMap ?? 'no map'); + deploymentStage(id, message, { phase: 'verify-map', checks, blocker: { code: 'map-mismatch', message, automatic: true, + href: `/servers/${encodeURIComponent(job.server)}?tab=config`, label: 'Map configuration' } }); continue; } - deploymentStage(id, 'Verified Komodo healthy, ' + (status.source === 'rcon' ? 'RCON healthy' : 'fresh telemetry') + ', and map ' + actualMap); - finishDeployment(id, { ok: true }); + db().transaction(() => { + deploymentStage(id, 'Verified Komodo healthy, ' + (status.source === 'rcon' ? 'RCON healthy' : 'fresh telemetry') + ', and map ' + actualMap, { + phase: 'verify-map', checks, blocker: null, checkpoints: [ + { key: 'container-action', label: 'Container startup verified' }, + { key: 'verify-container', label: 'Container readiness verified' }, + { key: 'verify-game', label: 'Game status verified' }, + { key: 'verify-map', label: 'Selected engine map verified' } + ] + }); + finishDeployment(id, { ok: true }); + })(); } catch (err) { - deploymentStage(id, 'Waiting for health verification: ' + (err as Error).message); + if (!stillCheckingCreation(id)) continue; + checks.push({ key: checks.length ? 'game' : 'container', label: checks.length ? 'Game status' : 'Container readiness', source: null, result: 'unknown', message: (err as Error).message }); + deploymentStage(id, 'Waiting for health verification: ' + (err as Error).message, { phase: checks.length > 1 ? 'verify-game' : 'verify-container', checks, + blocker: { code: 'observation-unavailable', message: (err as Error).message, automatic: true } }); } } @@ -790,7 +918,9 @@ async function runCreations(): Promise { const available = Math.max(0, fleetSettings().createConcurrency - queue.active.length); const eligible = queue.queued.filter(({ id }) => { const blocker = creationBlocker(id); - if (blocker) deploymentStage(id, blocker); + if (blocker) deploymentStage(id, blocker.message, { phase: 'startup-queue', blocker }); + else recordProgress(id, { phase: 'startup-queue', blocker: { code: available ? 'startup-queue' : 'startup-capacity', + message: available ? 'Eligible for startup admission in request order.' : 'Waiting for startup capacity to be released.', automatic: true } }); return !blocker; }); for (const { id } of eligible.slice(0, available)) { @@ -801,7 +931,7 @@ async function runCreations(): Promise { if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND action IN ('create', 'rebuild') AND state = 'queued'").get(id)) continue; const blocker = creationBlocker(id); if (blocker) { - deploymentStage(id, blocker); + deploymentStage(id, blocker.message, { phase: 'startup-queue', blocker }); continue; } try { outcomes.set(id, await provisionCreation(id)); } diff --git a/src/lib/server/provisioning/progress.ts b/src/lib/server/provisioning/progress.ts new file mode 100644 index 0000000..6c54282 --- /dev/null +++ b/src/lib/server/provisioning/progress.ts @@ -0,0 +1,120 @@ +import { db } from '../db/index.js'; +import { provisioningPhases, type ProvisioningPhase, type ProvisioningProgress, type ProvisioningCheck, + type ProvisioningCheckpoint, type ProvisioningBlocker, type RecoveryObservation } from '../../provisioning-detail.js'; + +export type CheckInput = Omit & + Partial>; +export type CheckpointInput = Omit & { at?: string; phase?: ProvisioningPhase | null }; +export interface ProgressUpdate { + phase?: ProvisioningPhase | null; + blocker?: ProvisioningBlocker | null; + /** A complete evaluation; omitted checks are no longer current evidence. */ + checks?: CheckInput[]; + checkpoint?: CheckpointInput; + checkpoints?: CheckpointInput[]; + command?: (Omit, 'at'> & { at?: string }) | null; + recovery?: (Omit & { at?: string }) | null; + recipeStatus?: ProvisioningProgress['recipeStatus']; + observedAt?: string | null; + event?: { kind: string; message: string; details?: Record }; +} + +const object = (value: unknown): value is Record => !!value && typeof value === 'object' && !Array.isArray(value); +const nullableString = (value: unknown) => value === null || typeof value === 'string'; +const phase = (value: unknown): value is ProvisioningPhase => provisioningPhases.includes(value as ProvisioningPhase); +const scalar = (value: unknown) => nullableString(value) || typeof value === 'number' && Number.isFinite(value); + +/** Invalid/old snapshots never manufacture evidence. Raw legacy stages remain available. */ +export function readProgress(row: { observation_json?: string | null }): ProvisioningProgress | null { + if (!row.observation_json) return null; + try { + const v: unknown = JSON.parse(row.observation_json); + if (!object(v) || v.version !== 1 || !(v.phase === null || phase(v.phase)) || !nullableString(v.phaseStartedAt) + || !nullableString(v.observedAt) || !Array.isArray(v.checks) || !Array.isArray(v.checkpoints)) return null; + if (v.blocker !== null && (!object(v.blocker) || typeof v.blocker.code !== 'string' || typeof v.blocker.message !== 'string' + || typeof v.blocker.automatic !== 'boolean' || (v.blocker.href !== undefined && !nullableString(v.blocker.href)) + || (v.blocker.label !== undefined && !nullableString(v.blocker.label)))) return null; + if (v.checks.length > 10 || !v.checks.every(c => object(c) && ['container', 'game', 'map', 'occupancy', 'stopped'].includes(String(c.key)) + && typeof c.label === 'string' && ['pass', 'waiting', 'fail', 'unknown'].includes(String(c.result)) + && (c.source === null || ['komodo', 'rcon', 'telemetry', 'combined'].includes(String(c.source))) + && typeof c.checkedAt === 'string' && nullableString(c.receivedAt) && scalar(c.expected) && scalar(c.actual) && nullableString(c.message))) return null; + if (v.checkpoints.length > 100 || !v.checkpoints.every(c => object(c) && typeof c.key === 'string' && typeof c.label === 'string' + && typeof c.at === 'string' && (c.phase === null || phase(c.phase)))) return null; + if (v.command !== null && (!object(v.command) || !['attempted', 'accepted', 'rejected', 'uncertain'].includes(String(v.command.state)) + || typeof v.command.at !== 'string' || typeof v.command.message !== 'string')) return null; + if (v.recovery !== null && (!object(v.recovery) || !['held', 'released', 'transferred', 'unknown'].includes(String(v.recovery.state)) + || typeof v.recovery.at !== 'string' || typeof v.recovery.message !== 'string' + || (v.recovery.operationId !== undefined && !nullableString(v.recovery.operationId)))) return null; + if (v.recipeStatus !== null && v.recipeStatus !== 'queued' && v.recipeStatus !== 'attempted') return null; + return v as unknown as ProvisioningProgress; + } catch { return null; } +} + +/** + * Records facts at the existing execution boundary, never performs a probe or command. + * Can be nested inside the caller's transaction with its state/legacy-stage writes. + */ +export function recordProgress(id: string, update: ProgressUpdate): void { + const handle = db(); + handle.transaction(() => { + const row = handle.prepare('SELECT observation_json FROM server_deployments WHERE id = ?').get(id) as + { observation_json: string | null } | undefined; + if (!row) return; + const now = new Date().toISOString(); + const previous = readProgress(row); + const next: ProvisioningProgress = previous ? { ...previous, checkpoints: [...previous.checkpoints] } : { + version: 1, phase: null, phaseStartedAt: null, observedAt: null, blocker: null, checks: [], checkpoints: [], + command: null, recovery: null, recipeStatus: null + }; + const events: { kind: string; message: string; details?: Record }[] = []; + if (update.phase !== undefined && update.phase !== next.phase) { + next.phase = update.phase; + next.phaseStartedAt = update.phase === null ? null : now; + next.blocker = null; + } + next.observedAt = update.observedAt === undefined ? now : update.observedAt; + if (update.blocker !== undefined) { + const old = next.blocker; + next.blocker = update.blocker; + // Counts/error wording can change on every poll without a new kind of hold. + if (update.blocker && (old?.code !== update.blocker.code || old?.href !== update.blocker.href)) { + events.push({ kind: 'blocker', message: update.blocker.message, + details: { code: update.blocker.code, href: update.blocker.href ?? null } }); + } + } + if (update.checks !== undefined) { + next.checks = update.checks.map(check => ({ ...check, checkedAt: check.checkedAt ?? now, + receivedAt: check.receivedAt ?? null, expected: check.expected ?? null, actual: check.actual ?? null, + message: check.message ?? null })); + } + for (const input of [...(update.checkpoints ?? []), ...(update.checkpoint ? [update.checkpoint] : [])]) { + if (next.checkpoints.some(c => c.key === input.key)) continue; + const point: ProvisioningCheckpoint = { ...input, at: input.at ?? now, phase: input.phase ?? (phase(input.key) ? input.key : next.phase) }; + next.checkpoints.push(point); + events.push({ kind: 'checkpoint', message: point.label, details: { key: point.key, at: point.at } }); + } + if (update.command !== undefined) { + if (!update.command) next.command = null; + else if (next.command?.state !== update.command.state || next.command?.message !== update.command.message) { + next.command = { ...update.command, at: update.command.at ?? now }; + events.push({ kind: 'command', message: next.command.message, details: { state: next.command.state, at: next.command.at } }); + } + } + if (update.recovery !== undefined) { + if (!update.recovery) next.recovery = null; + else { + const changed = next.recovery?.state !== update.recovery.state || next.recovery?.operationId !== update.recovery.operationId; + next.recovery = { ...update.recovery, at: changed ? update.recovery.at ?? now : next.recovery!.at }; + if (changed) events.push({ kind: 'recovery', message: next.recovery.message, + details: { state: next.recovery.state, at: next.recovery.at, operationId: next.recovery.operationId ?? null } }); + } + } + if (update.recipeStatus !== undefined) next.recipeStatus = update.recipeStatus; + if (update.event) events.push(update.event); + handle.prepare(`UPDATE server_deployments SET phase_key = ?, phase_started_at = ?, observed_at = ?, observation_json = ? WHERE id = ?`) + .run(next.phase, next.phaseStartedAt, next.observedAt, JSON.stringify(next), id); + const insert = handle.prepare(`INSERT INTO server_deployment_events (deployment_id, created_at, message, kind, phase_key, details_json) + VALUES (?, ?, ?, ?, ?, ?)`); + for (const event of events) insert.run(id, now, event.message, event.kind, next.phase, event.details ? JSON.stringify(event.details) : null); + })(); +} diff --git a/src/lib/server/provisioning/requests.ts b/src/lib/server/provisioning/requests.ts index 9e14149..76a9569 100644 --- a/src/lib/server/provisioning/requests.ts +++ b/src/lib/server/provisioning/requests.ts @@ -6,6 +6,7 @@ import { beginDeployment, deploymentStage, finishDeployment, recordDeploymentEve import { queueRebuildServer } from './instances.js'; import { restartRequestInFlight } from './restarts.js'; import { runtimeActions, type RuntimeAction } from '../../provisioning.js'; +import { recordProgress } from './progress.js'; export interface LifecycleOperation { id: string; server: string; action: string; state: string; stage: string; @@ -67,7 +68,8 @@ export function requestRestart(server: string, source: RestartSource): Lifecycle if (!id) { id = beginDeployment(server, 'restart', `${source.kind}:${source.id}`); db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id); - deploymentStage(id, 'Waiting to verify the server is empty'); + deploymentStage(id, 'Waiting to verify the server is empty', { phase: 'eligibility', observedAt: null, + blocker: { code: 'eligibility', message: 'Waiting for eligibility and a fresh empty or stopped server.', automatic: true } }); if (row.restarted_at) adoptSubmittedRestart(id, server, row.restarted_at); } db().prepare('INSERT OR IGNORE INTO provisioning_sources (operation_id, kind, source_id, server) VALUES (?, ?, ?, ?)') @@ -79,12 +81,17 @@ export function requestRestart(server: string, source: RestartSource): Lifecycle /** Upgrade observation only: existing restart intent is never resubmitted. */ function adoptSubmittedRestart(id: string, server: string, submittedAt: string): void { - const slot = db().prepare('SELECT operation_id, requested_at, healthy_at, players FROM restart_rollout WHERE server = ?').get(server) as - { operation_id: string | null; requested_at: string; healthy_at: string | null; players: number | null } | undefined; + const slot = db().prepare('SELECT operation_id, requested_at, checked_at, healthy_at, players FROM restart_rollout WHERE server = ?').get(server) as + { operation_id: string | null; requested_at: string; checked_at: string | null; healthy_at: string | null; players: number | null } | undefined; db().prepare("UPDATE server_deployments SET state = 'awaiting-health', submitted_at = ? WHERE id = ?").run(submittedAt, id); + recordProgress(id, { command: { state: 'attempted', at: submittedAt, message: 'Adopted a previously recorded submission; no command is replayed.' } }); if (slot && ((slot.operation_id && slot.requested_at >= submittedAt) || (slot.requested_at < submittedAt && !slot.healthy_at))) { + const finishedAt = new Date().toISOString(); + const error = 'Existing restart cannot be correlated safely. Inspect the server before retrying.'; db().prepare("UPDATE server_deployments SET state = 'interrupted', finished_at = ?, error = ? WHERE id = ?") - .run(new Date().toISOString(), 'Existing restart cannot be correlated safely. Inspect the server before retrying.', id); + .run(finishedAt, error, id); + recordProgress(id, { recovery: { state: 'unknown', message: 'Existing restart recovery could not be correlated with this operation.', operationId: id }, + observedAt: null, event: { kind: 'outcome', message: error, details: { state: 'interrupted', finishedAt, error } } }); return; } if (slot && slot.requested_at >= submittedAt) { @@ -96,16 +103,24 @@ function adoptSubmittedRestart(id: string, server: string, submittedAt: string): } if (slot?.healthy_at && slot.requested_at >= submittedAt) { db().prepare("UPDATE server_deployments SET state = 'done', finished_at = ?, players = ? WHERE id = ?").run(slot.healthy_at, slot.players, id); + recordProgress(id, { observedAt: slot.healthy_at, event: { kind: 'outcome', message: 'Adopted recorded recovery completion.', + details: { state: 'done', finishedAt: slot.healthy_at, error: null, warnings: [] } } }); } - deploymentStage(id, 'Adopted existing restart; verifying recovery without restarting again'); + deploymentStage(id, 'Adopted existing restart; verifying recovery without restarting again', { + phase: 'verify-container', checks: [], blocker: null, observedAt: slot?.checked_at ?? null, + recovery: { state: slot?.healthy_at && slot.requested_at >= submittedAt ? 'released' : 'held', + at: slot?.healthy_at && slot.requested_at >= submittedAt ? slot.healthy_at : undefined, + message: slot?.healthy_at && slot.requested_at >= submittedAt ? 'Adopted recorded recovery completion.' : 'Observing the previously submitted restart without replaying it.', operationId: id } + }); } export function cancelProvisioning(id: string, reason = 'Cancelled by requester'): boolean { return db().transaction(() => { + const finishedAt = new Date().toISOString(); const result = db().prepare(`UPDATE server_deployments SET state = 'cancelled', finished_at = ?, stage = ?, error = NULL WHERE id = ? AND action IN ('start','restart','stop','rebuild') AND state = 'waiting' AND submitted_at IS NULL`) - .run(new Date().toISOString(), reason, id); - if (result.changes) recordDeploymentEvent(id, reason); + .run(finishedAt, reason, id); + if (result.changes) recordProgress(id, { blocker: null, checks: [], event: { kind: 'outcome', message: reason, details: { state: 'cancelled', finishedAt, error: null, warnings: [] } } }); return result.changes > 0; })(); } @@ -143,7 +158,8 @@ export function requestLifecycle(server: string, action: RuntimeAction) { const id = beginDeployment(server, action); db().prepare('UPDATE servers SET desired_state = ? WHERE name = ?').run(action === 'stop' ? 'stopped' : 'running', server); db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id); - deploymentStage(id, action === 'start' ? 'Waiting to start the container' : 'Waiting to verify the server is empty'); + deploymentStage(id, action === 'start' ? 'Waiting to start the container' : 'Waiting to verify the server is empty', { phase: 'eligibility', observedAt: null, + blocker: { code: 'eligibility', message: action === 'start' ? 'Waiting for start eligibility and recovery capacity.' : 'Waiting for eligibility and a fresh empty or stopped server.', automatic: true } }); return { ok: true as const, deploymentId: id }; })(); } @@ -177,7 +193,8 @@ function queueWatchdogAction(server: string, action: 'restart' | 'rebuild', inci } else { id = beginDeployment(server, action, `watchdog:${incidentId}`); db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id); - deploymentStage(id, 'Waiting to verify the server is empty'); + deploymentStage(id, 'Waiting to verify the server is empty', { phase: 'eligibility', observedAt: null, + blocker: { code: 'eligibility', message: 'Waiting for eligibility and a fresh empty or stopped server.', automatic: true } }); } db().prepare("INSERT INTO provisioning_sources (operation_id, kind, source_id, server) VALUES (?, 'watchdog', ?, ?)").run(id, incidentId, server); return { ok: true, deploymentId: id }; @@ -209,8 +226,14 @@ export function requestWatchdogLifecycle(server: string, action: 'restart' | 're WHERE r.server = ? AND r.healthy_at IS NULL AND d.state IN ('failed','interrupted','cancelled') AND a.cycle < i.cycle`).get(incidentId, server) as { operation_id: string } | undefined; if (retained && !restartRequestInFlight(server)) { - db().prepare('UPDATE restart_rollout SET operation_id = ? WHERE server = ? AND operation_id = ? AND healthy_at IS NULL') + const transferred = db().prepare('UPDATE restart_rollout SET operation_id = ? WHERE server = ? AND operation_id = ? AND healthy_at IS NULL') .run(next.deploymentId, server, retained.operation_id); + if (transferred.changes) { + recordProgress(retained.operation_id, { recovery: { state: 'transferred', operationId: next.deploymentId, + message: 'Recovery capacity transferred to the explicit new incident attempt.' } }); + recordProgress(next.deploymentId, { recovery: { state: 'held', operationId: next.deploymentId, + message: 'Recovery capacity retained from the prior incident attempt.' } }); + } recordDeploymentEvent(retained.operation_id, 'Recovery capacity handed to explicit Watchdog retry ' + next.deploymentId); recordDeploymentEvent(next.deploymentId, 'Retained recovery capacity from prior incident attempt ' + retained.operation_id); } @@ -249,8 +272,16 @@ export function escalateWatchdogRestart(incidentId: string, operationId: string, const reason = 'Watchdog restart recovery timed out; handed recovery to rebuild ' + replacement.deploymentId; if (operation.state === 'awaiting-health') finishDeployment(operationId, { ok: false, message: reason }); else recordDeploymentEvent(operationId, reason); - if (slot) db().prepare('UPDATE restart_rollout SET operation_id = ?, error = ? WHERE server = ? AND operation_id = ? AND healthy_at IS NULL') - .run(replacement.deploymentId, reason, operation.server, operationId); + if (slot) { + const transferred = db().prepare('UPDATE restart_rollout SET operation_id = ?, error = ? WHERE server = ? AND operation_id = ? AND healthy_at IS NULL') + .run(replacement.deploymentId, reason, operation.server, operationId); + if (transferred.changes) { + recordProgress(operationId, { recovery: { state: 'transferred', operationId: replacement.deploymentId, + message: 'Recovery capacity transferred to the rebuild operation.' } }); + recordProgress(replacement.deploymentId, { recovery: { state: 'held', operationId: replacement.deploymentId, + message: 'Recovery capacity retained from the previous restart.' } }); + } + } recordDeploymentEvent(replacement.deploymentId, 'Recovery handed off from restart ' + operationId + '; waiting for a fresh empty or stopped server'); return replacement; })(); diff --git a/src/lib/server/provisioning/restarts.ts b/src/lib/server/provisioning/restarts.ts index 515635b..913475d 100644 --- a/src/lib/server/provisioning/restarts.ts +++ b/src/lib/server/provisioning/restarts.ts @@ -5,6 +5,8 @@ import { probeServerStatus } from '../fleet/status-evidence.js'; import { deploymentStage, finishDeployment } from './history.js'; import { applyStagedFilesForOperation } from '../fleet/staged-files.js'; import type { RuntimeAction } from '../../provisioning.js'; +import type { ProvisioningBlocker } from '../../provisioning-detail.js'; +import { recordProgress, type ProgressUpdate } from './progress.js'; // Komodo queues restart work. Don't mistake the still-running old container // for a completed restart when an API kick immediately follows the request. @@ -38,21 +40,28 @@ export function restartSettling(server: string): boolean { } export function rolloutWaitReason(server: string, operationId = '', action: RuntimeAction = 'restart'): string | null { - if (db().prepare(`SELECT 1 FROM server_deployments WHERE server = ? AND id != ? AND server_deleted_at IS NULL + return rolloutBlocker(server, operationId, action)?.message ?? null; +} + +export function rolloutBlocker(server: string, operationId = '', action: RuntimeAction = 'restart'): ProvisioningBlocker | null { + const conflict = db().prepare(`SELECT id FROM server_deployments WHERE server = ? AND id != ? AND server_deleted_at IS NULL AND ((action IN ('start','stop','rebuild') AND state IN ('running','awaiting-health')) - OR (action = 'destroy' AND state IN ('waiting','running')))`).get(server, operationId)) { - return 'Waiting for the current provisioning operation'; + OR (action = 'destroy' AND state IN ('waiting','running')))`).get(server, operationId) as { id: string } | undefined; + if (conflict) { + return { code: 'operation-conflict', message: 'Waiting for the current provisioning operation', automatic: true, + href: `/provisioning/${encodeURIComponent(conflict.id)}`, label: 'Blocking operation' }; } const slot = slotFor(server); const ownHandoff = slot?.operation_id === operationId && !!db().prepare(`SELECT 1 FROM server_deployments d WHERE d.id = ? AND d.server_deleted_at IS NULL AND d.action IN ('restart','rebuild') AND EXISTS ( SELECT 1 FROM provisioning_sources s WHERE s.operation_id = d.id AND s.kind = 'watchdog')`).get(operationId); if (slot?.healthy_at === null && !ownHandoff) { - return `Waiting for this server to recover from its previous operation${slot.error ? `: ${slot.error}` : ''}`; + return { code: 'previous-recovery', message: `Waiting for this server to recover from its previous operation${slot.error ? `: ${slot.error}` : ''}`, automatic: true, + href: slot.operation_id ? `/provisioning/${encodeURIComponent(slot.operation_id)}` : null, label: 'Previous operation' }; } const { n } = db().prepare('SELECT COUNT(*) AS n FROM restart_rollout WHERE healthy_at IS NULL').get() as { n: number }; return n - (ownHandoff && slot?.healthy_at === null ? 1 : 0) >= fleetSettings().restartConcurrency - ? `Queued to ${action}; waiting for another server to recover` : null; + ? { code: 'restart-capacity', message: `Queued to ${action}; waiting for another server to recover`, automatic: true } : null; } export type RolloutRestartResult = @@ -71,6 +80,7 @@ export async function restartInRollout(server: string, options: { operationId: s ON CONFLICT(server) DO UPDATE SET requested_at = excluded.requested_at, checked_at = NULL, healthy_at = NULL, players = NULL, error = NULL, operation_id = excluded.operation_id`) .run(server, new Date().toISOString(), operationId); + recordProgress(operationId, { blocker: null, recovery: { state: 'held', message: 'Restart recovery capacity reserved.', operationId } }); return null; }).immediate(); if (waitReason) return { kind: 'queued', message: waitReason }; @@ -81,12 +91,20 @@ export async function restartInRollout(server: string, options: { operationId: s applyStagedFilesForOperation(operationId, server); } catch (err) { const message = `File activation failed: ${(err as Error).message}`; - db().prepare('DELETE FROM restart_rollout WHERE server = ? AND operation_id = ?').run(server, operationId); - finishDeployment(operationId, { ok: false, message }); + db().transaction(() => { + db().prepare('DELETE FROM restart_rollout WHERE server = ? AND operation_id = ?').run(server, operationId); + recordProgress(operationId, { recovery: { state: 'released', message: 'Capacity released before any remote command: file activation failed.', operationId } }); + finishDeployment(operationId, { ok: false, message }); + })(); return { kind: 'failed', message }; } - db().prepare('UPDATE server_deployments SET submitted_at = ? WHERE id = ?').run(new Date().toISOString(), operationId); - deploymentStage(operationId, options.action === 'start' ? 'Starting container' : 'Restarting container'); + db().transaction(() => { + db().prepare('UPDATE server_deployments SET submitted_at = ? WHERE id = ?').run(new Date().toISOString(), operationId); + deploymentStage(operationId, options.action === 'start' ? 'Starting container' : 'Restarting container', { + phase: 'container-action', blocker: null, checks: [], checkpoint: { key: 'eligibility', label: 'Eligibility checks passed' }, + command: { state: 'attempted', message: `${options.action === 'start' ? 'Start' : 'Restart'} submission attempted.` } + }); + })(); requestsInFlight.set(server, operationId); db().prepare(`UPDATE servers SET container_state = 'restarting', last_error = NULL, last_reply_snippet = NULL, game_started_at = NULL, game_ended_at = NULL WHERE name = ?`).run(server); @@ -94,24 +112,38 @@ export async function restartInRollout(server: string, options: { operationId: s const result = options.action === 'start' ? await komodoRuntimeAction(server, 'start') : await komodoRestart(server); if (!db().prepare('SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL').get(operationId)) return { kind: 'failed', message: 'Original server was deleted' }; if (result.ok) { - db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ? AND server_deleted_at IS NULL").run(operationId); - deploymentStage(operationId!, 'Waiting for container and game health (RCON or fresh telemetry)'); + db().transaction(() => { + db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ? AND server_deleted_at IS NULL").run(operationId); + deploymentStage(operationId!, 'Waiting for container and game health (RCON or fresh telemetry)', { + phase: 'verify-container', command: { state: 'accepted', message: 'Komodo accepted the request; recovery still requires verification.' }, + checkpoint: { key: 'container-action', label: 'Container command accepted' }, + blocker: { code: 'recovery-settling', message: 'Waiting for the recovery settling interval before checking health.', automatic: true } + }); + })(); return { kind: 'started' }; } // These failures cannot have queued a remote restart. Other failures // (especially a timeout) may have: retain capacity until recovery. - if (result.kind === 'no-name' || result.kind === 'unconfigured') { - db().prepare('DELETE FROM restart_rollout WHERE server = ? AND operation_id = ?').run(server, operationId); - } else { - db().prepare('UPDATE restart_rollout SET error = ? WHERE server = ? AND operation_id = ?').run(`komodo: ${result.message}`, server, operationId); - } - finishDeployment(operationId!, { ok: false, message: `komodo: ${result.message}` }); + db().transaction(() => { + if (result.kind === 'no-name' || result.kind === 'unconfigured') { + db().prepare('DELETE FROM restart_rollout WHERE server = ? AND operation_id = ?').run(server, operationId); + recordProgress(operationId, { command: { state: 'rejected', message: 'The command could not be submitted: ' + result.message }, + recovery: { state: 'released', message: 'Capacity released because no remote command could be submitted.', operationId } }); + } else { + db().prepare('UPDATE restart_rollout SET error = ? WHERE server = ? AND operation_id = ?').run(`komodo: ${result.message}`, server, operationId); + recordProgress(operationId, { command: { state: 'uncertain', message: 'The command may have reached Komodo: ' + result.message } }); + } + finishDeployment(operationId!, { ok: false, message: `komodo: ${result.message}` }); + })(); return { kind: 'failed', message: `komodo: ${result.message}` }; } catch (err) { if (!db().prepare('SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL').get(operationId)) return { kind: 'failed', message: 'Original server was deleted' }; const message = `komodo: ${(err as Error).message}`; - db().prepare('UPDATE restart_rollout SET error = ? WHERE server = ? AND operation_id = ?').run(message, server, operationId); - finishDeployment(operationId!, { ok: false, message }); + db().transaction(() => { + db().prepare('UPDATE restart_rollout SET error = ? WHERE server = ? AND operation_id = ?').run(message, server, operationId); + recordProgress(operationId, { command: { state: 'uncertain', message: 'The command may have reached Komodo: ' + (err as Error).message } }); + finishDeployment(operationId!, { ok: false, message }); + })(); return { kind: 'failed', message }; } finally { // Start the grace period at completion, including slow/failed requests. @@ -145,31 +177,47 @@ async function refresh(): Promise { if (Date.now() - Date.parse(slot.checked_at ?? slot.requested_at) < HEALTH_POLL_MS) continue; let error: string | null = null; let players: number | null = null; + const checks: NonNullable = []; try { const health = await komodoInspectHealthy(slot.server); + checks.push({ key: 'container', label: 'Container readiness', result: health.ok ? 'pass' : 'waiting', source: 'komodo', actual: health.state ?? null, + checkedAt: new Date().toISOString(), message: health.ok ? null : health.reason }); if (!health.ok) { error = `unhealthy: ${health.reason}${health.snippet ? ` snippet=${health.snippet.slice(0, 400)}` : ''}`; } else { const status = await probeServerStatus(slot.server, slot.requested_at); + checks.push({ key: 'game', label: 'Game status', result: status.ok ? 'pass' : 'waiting', source: status.ok ? status.source : null, + checkedAt: status.observedAt, receivedAt: status.ok ? status.receivedAt : null, actual: status.ok ? status.players : null, + message: status.ok ? null : status.reason }); if (!status.ok) error = `rcon not answering and no fresh telemetry: ${status.reason}`; else players = status.players; } } catch (err) { error = `health check: ${(err as Error).message}`; + checks.push({ key: checks.length ? 'game' : 'container', label: checks.length ? 'Game status' : 'Container readiness', result: 'unknown', source: null, message: error }); } const now = new Date().toISOString(); - const checked = db().prepare(`UPDATE restart_rollout SET checked_at = ?, healthy_at = ?, players = ?, error = ? - WHERE server = ? AND requested_at = ? AND operation_id IS ? AND healthy_at IS NULL`) - .run(now, error ? null : now, players, error, slot.server, slot.requested_at, slot.operation_id); - if (!checked.changes) continue; - if (slot.operation_id && db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'awaiting-health'").get(slot.operation_id)) { - db().prepare('UPDATE server_deployments SET checked_at = ?, players = ? WHERE id = ?').run(now, players, slot.operation_id); - if (error) deploymentStage(slot.operation_id, 'Waiting for health: ' + error); - else { - deploymentStage(slot.operation_id, 'Verified container and game health (RCON or fresh telemetry)'); - finishDeployment(slot.operation_id, { ok: true }); + db().transaction(() => { + const checked = db().prepare(`UPDATE restart_rollout SET checked_at = ?, healthy_at = ?, players = ?, error = ? + WHERE server = ? AND requested_at = ? AND operation_id IS ? AND healthy_at IS NULL`) + .run(now, error ? null : now, players, error, slot.server, slot.requested_at, slot.operation_id); + if (!checked.changes) return; + if (slot.operation_id) recordProgress(slot.operation_id, { + phase: checks.some(check => check.key === 'game') ? 'verify-game' : 'verify-container', checks, observedAt: now, + blocker: error ? { code: 'recovery-health', message: error, automatic: true } : null, + recovery: { state: error ? 'held' : 'released', at: now, operationId: slot.operation_id, + message: error ? 'Restart capacity remains reserved while recovery is checked.' : 'Container and game recovery verified; restart capacity released.' }, + checkpoints: error ? [] : [{ key: 'verify-container', label: 'Container readiness verified', at: now }, { key: 'verify-game', label: 'Game status verified', at: now }] + }); + if (slot.operation_id && db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'awaiting-health'").get(slot.operation_id)) { + db().prepare('UPDATE server_deployments SET checked_at = ?, players = ? WHERE id = ?').run(now, players, slot.operation_id); + if (error) deploymentStage(slot.operation_id, 'Waiting for health: ' + error); + else { + deploymentStage(slot.operation_id, 'Verified container and game health (RCON or fresh telemetry)'); + finishDeployment(slot.operation_id, { ok: true }); + } } - } + })(); } } diff --git a/src/lib/server/provisioning/worker.ts b/src/lib/server/provisioning/worker.ts index 5e8306b..33e720c 100644 --- a/src/lib/server/provisioning/worker.ts +++ b/src/lib/server/provisioning/worker.ts @@ -5,10 +5,12 @@ import { getServer } from '../fleet/servers.js'; import { deploymentStage, finishDeployment, REMOVING_SERVER } from './history.js'; import { processCreations, processDestructions, rebuildServer } from './instances.js'; import { komodoInspectHealthy, komodoRuntimeAction } from '../integrations/komodo.js'; -import { probePlayerCount } from '../fleet/status-evidence.js'; -import { refreshRestartRollout, restartInRollout, rolloutWaitReason } from './restarts.js'; +import { probePlayerEvidence } from '../fleet/status-evidence.js'; +import { refreshRestartRollout, restartInRollout, rolloutBlocker } from './restarts.js'; import { cancelProvisioning } from './requests.js'; import type { RuntimeAction } from '../../provisioning.js'; +import type { ProvisioningBlocker } from '../../provisioning-detail.js'; +import { recordProgress } from './progress.js'; const stoppedStates = new Set(['stopped', 'exited', 'created', 'dead']); export const isStopped = (state?: string) => stoppedStates.has(state?.toLowerCase() ?? ''); @@ -18,7 +20,7 @@ const sourcesFor = (id: string) => db().prepare('SELECT kind, source_id FROM pro const priority = (sources: Source[]) => sources.some(s => s.kind === 'watchdog') ? 2 : sources.some(s => s.kind === 'beta') ? 0 : sources.some(s => s.kind === 'campaign') ? 1 : sources.some(s => s.kind === 'maintenance') ? 3 : 2; /** Source work may hold or withdraw a request, but never executes it. */ -function sourceEligibility(operation: Operation): 'ready' | 'held' | 'cancelled' { +function sourceEligibility(operation: Operation, onHold?: (source: Source) => void): 'ready' | 'held' | 'cancelled' { const sources = sourcesFor(operation.id); if (!sources.length) return 'ready'; // Watchdog intent gates every linked source, including its drift campaign. @@ -26,7 +28,7 @@ function sourceEligibility(operation: Operation): 'ready' | 'held' | 'cancelled' if (!db().prepare("SELECT 1 FROM watchdog_incidents WHERE id = ? AND server = ? AND server_deleted_at IS NULL AND state IN ('open','recovering')").get(source.source_id, operation.server) || getServer(operation.server)?.desired_state === 'stopped') return 'cancelled'; const incident = db().prepare('SELECT kind FROM watchdog_incidents WHERE id = ?').get(source.source_id) as { kind: 'health' | 'drift' }; - if (!maintenanceAllowed(incident.kind)) return 'held'; + if (!maintenanceAllowed(incident.kind)) { onHold?.(source); return 'held'; } } let held = false; let ready = false; @@ -36,7 +38,7 @@ function sourceEligibility(operation: Operation): 'ready' | 'held' | 'cancelled' .get(source.source_id, operation.server) as { restart_state: string; files_changed: number; provisioning_id: string } | undefined; if (row?.provisioning_id !== operation.id || !row.files_changed) continue; if (['pending','failed'].includes(row.restart_state)) ready = true; - if (['held','manual'].includes(row.restart_state)) held = true; + if (['held','manual'].includes(row.restart_state)) { held = true; onHold?.(source); } } else if (source.kind === 'beta') { const row = db().prepare(`SELECT b.state, bs.stage, bs.provisioning_id, p.outcome FROM beta_servers bs JOIN betas b ON b.id = bs.beta_id LEFT JOIN beta_test_plans p ON p.beta_id = b.id WHERE bs.beta_id = ? AND bs.server = ? AND bs.server_deleted_at IS NULL`) @@ -44,12 +46,12 @@ function sourceEligibility(operation: Operation): 'ready' | 'held' | 'cancelled' if (row?.provisioning_id !== operation.id || !['running','ready-promote'].includes(row.state)) continue; if (!['awaiting-empty','restarting','awaiting-health'].includes(row.stage)) continue; if (!row.outcome || ['active','passed'].includes(row.outcome)) ready = true; - if (row.outcome === 'paused') held = true; + if (row.outcome === 'paused') { held = true; onHold?.(source); } } else if (source.kind === 'maintenance') { const row = db().prepare('SELECT state, provisioning_id, kind, origin FROM maintenance_jobs WHERE id = ? AND server_deleted_at IS NULL').get(source.source_id) as { state: string; provisioning_id: string; kind: string; origin: string } | undefined; if (row?.provisioning_id === operation.id && ['queued','restarting'].includes(row.state)) { if (getServer(operation.server)?.desired_state === 'stopped') return 'cancelled'; - if (!maintenanceAllowed() || (row.kind === 'update' && row.origin === 'automatic' && !maintenanceAllowed('updates'))) return 'held'; + if (!maintenanceAllowed() || (row.kind === 'update' && row.origin === 'automatic' && !maintenanceAllowed('updates'))) { onHold?.(source); return 'held'; } ready = true; } } else ready = true; @@ -57,26 +59,31 @@ function sourceEligibility(operation: Operation): 'ready' | 'held' | 'cancelled' return held ? 'held' : ready ? 'ready' : 'cancelled'; } -function blocker(server: string, id: string, action: RuntimeAction): string | null { +function blocker(server: string, id: string, action: RuntimeAction): ProvisioningBlocker | null { const sources = sourcesFor(id); const rank = priority(sources); const others = db().prepare(`SELECT id, state, action, rowid FROM server_deployments WHERE server = ? AND id != ? AND server_deleted_at IS NULL AND state IN ('queued','waiting','running','awaiting-health')`).all(server, id) as { id: string; state: string; action: string; rowid: number }[]; const own = db().prepare('SELECT rowid FROM server_deployments WHERE id = ?').get(id) as { rowid: number }; - if (others.some(other => other.state !== 'waiting' || other.action === 'destroy' - || priority(sourcesFor(other.id)) < rank || (priority(sourcesFor(other.id)) === rank && other.rowid < own.rowid))) { - return 'Waiting for another provisioning operation on this server'; + const conflict = others.find(other => other.state !== 'waiting' || other.action === 'destroy' + || priority(sourcesFor(other.id)) < rank || (priority(sourcesFor(other.id)) === rank && other.rowid < own.rowid)); + if (conflict) { + return { code: 'operation-conflict', message: 'Waiting for another provisioning operation on this server', automatic: true, + href: `/provisioning/${encodeURIComponent(conflict.id)}`, label: 'Blocking operation' }; } // Beta activation must not wait on the campaign rows that the beta itself // holds. Campaign/maintenance callers likewise never block on their own job. - if (rank > 0 && db().prepare(`SELECT 1 FROM beta_servers bs JOIN betas b ON b.id = bs.beta_id - WHERE bs.server = ? AND bs.server_deleted_at IS NULL AND b.state IN ('running','ready-promote') LIMIT 1`).get(server)) return 'Waiting for beta work on this server'; - if (rank > 1 && db().prepare(`SELECT 1 FROM deployment_servers ds WHERE ds.server = ? AND ds.server_deleted_at IS NULL + const beta = rank > 0 ? db().prepare(`SELECT b.id FROM beta_servers bs JOIN betas b ON b.id = bs.beta_id + WHERE bs.server = ? AND bs.server_deleted_at IS NULL AND b.state IN ('running','ready-promote') LIMIT 1`).get(server) as { id: string } | undefined : undefined; + if (beta) return { code: 'beta-work', message: 'Waiting for beta work on this server', automatic: true, href: `/betas/${encodeURIComponent(beta.id)}`, label: 'Blocking beta' }; + const campaign = rank > 1 ? db().prepare(`SELECT ds.deployment_id FROM deployment_servers ds WHERE ds.server = ? AND ds.server_deleted_at IS NULL AND ds.restart_state IN ('pending','failed','held') AND (ds.provisioning_id IS NULL OR EXISTS ( SELECT 1 FROM server_deployments d WHERE d.id = ds.provisioning_id AND d.state IN ('queued','waiting','running','awaiting-health'))) AND NOT EXISTS (SELECT 1 FROM watchdog_actions a WHERE a.operation_id = ? AND a.campaign_id = ds.deployment_id) - LIMIT 1`).get(server, id)) return 'Waiting for campaign work on this server'; - return rolloutWaitReason(server, id, action); + LIMIT 1`).get(server, id) as { deployment_id: string } | undefined : undefined; + if (campaign) return { code: 'campaign-work', message: 'Waiting for campaign work on this server', automatic: true, + href: `/campaigns/${encodeURIComponent(campaign.deployment_id)}`, label: 'Blocking campaign' }; + return rolloutBlocker(server, id, action); } interface Operation { id: string; server: string; action: RuntimeAction; state: string; stage: string } @@ -100,7 +107,9 @@ async function run(): Promise { try { await processOperation(operation); } catch (err) { // Observation failures are retried; remote mutations are never replayed. - deploymentStage(operation.id, 'Waiting for verification: ' + (err as Error).message); + deploymentStage(operation.id, 'Waiting for verification: ' + (err as Error).message, { + checks: [], blocker: { code: 'observation-unavailable', message: (err as Error).message, automatic: true } + }); } } } @@ -112,36 +121,65 @@ async function processOperation(operation: Operation): Promise { // Watchdog is paused; never strand the reserved replacement identity. if (action === 'rebuild' && operation.stage === REMOVING_SERVER) { await rebuildServer(id); return; } if (operation.state === 'waiting') { - const eligibility = sourceEligibility(operation); + const heldBy: Source[] = []; + const eligibility = sourceEligibility(operation, source => heldBy.push(source)); if (eligibility === 'cancelled') { cancelProvisioning(id, 'Requester cancelled or replaced this operation'); return; } - if (eligibility === 'held') { deploymentStage(id, 'Waiting for the requester to resume'); return; } + if (eligibility === 'held') { + const source = heldBy[0]; + const href = source ? source.kind === 'campaign' ? `/campaigns/${encodeURIComponent(source.source_id)}` + : source.kind === 'beta' ? `/betas/${encodeURIComponent(source.source_id)}` + : source.kind === 'watchdog' ? `/maintenance/${encodeURIComponent(source.source_id)}` + : `/maintenance?view=activity#job-${encodeURIComponent(source.source_id)}` : null; + deploymentStage(id, 'Waiting for the requester to resume', { phase: 'eligibility', checks: [], + blocker: { code: 'requester-held', message: 'Waiting for the requester to resume', automatic: false, href, label: 'Requesting work' } }); + return; + } } if (!getServer(server)) { finishDeployment(id, { ok: false, message: 'Server is no longer registered.' }); return; } const watchdog = sourcesFor(id).some(source => source.kind === 'watchdog'); - if (!komodoConfigured() || (!rconConfigured() && !watchdog)) { deploymentStage(id, 'Waiting for Komodo and RCON configuration'); return; } + if (!komodoConfigured() || (!rconConfigured() && !watchdog)) { deploymentStage(id, 'Waiting for Komodo and RCON configuration', { + phase: 'eligibility', checks: [], blocker: { code: 'configuration', message: 'Waiting for Komodo and RCON configuration', automatic: false, href: '/settings', label: 'Integration settings' } + }); return; } if (operation.state === 'awaiting-health') { const health = await komodoInspectHealthy(server); + if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'awaiting-health'").get(id)) return; + recordProgress(id, { phase: 'verify-stopped', checks: [{ key: 'stopped', label: 'Container stopped', source: 'komodo', + result: isStopped(health.state) ? 'pass' : health.state ? 'waiting' : 'unknown', actual: health.state ?? null, + message: health.ok ? null : health.reason }], blocker: isStopped(health.state) ? null + : { code: 'stopped-state', message: 'Waiting for an explicit stopped container state.', automatic: true } }); if (isStopped(health.state)) { - db().prepare("UPDATE servers SET container_state = ?, last_players = NULL, last_roster = NULL, game_started_at = NULL, game_ended_at = NULL WHERE name = ?").run(health.state, server); - deploymentStage(id, 'Verified container stopped'); - finishDeployment(id, { ok: true }); + db().transaction(() => { + db().prepare("UPDATE servers SET container_state = ?, last_players = NULL, last_roster = NULL, game_started_at = NULL, game_ended_at = NULL WHERE name = ?").run(health.state, server); + deploymentStage(id, 'Verified container stopped', { checkpoint: { key: 'verify-stopped', label: 'Stopped container state verified' } }); + finishDeployment(id, { ok: true }); + })(); } else deploymentStage(id, 'Waiting for container to stop' + (!health.ok ? ': ' + health.reason : '')); return; } const reason = blocker(server, id, action); - if (reason) { deploymentStage(id, reason); return; } + if (reason) { deploymentStage(id, reason.message, { phase: 'eligibility', checks: [], blocker: reason }); return; } const health = await komodoInspectHealthy(server); if (action !== 'start' && !isStopped(health.state)) { - const players = await probePlayerCount(server); + const evidence = await probePlayerEvidence(server); + if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'waiting'").get(id)) return; + const players = evidence.players; + recordProgress(id, { phase: 'eligibility', checks: [{ key: 'occupancy', label: 'Player occupancy', result: players === 0 ? 'pass' : players === null ? 'unknown' : 'waiting', + source: evidence.source, checkedAt: evidence.observedAt, receivedAt: evidence.receivedAt, expected: 0, actual: players }], + blocker: players === 0 ? null : { code: players === null ? 'players-unknown' : 'players-present', + message: players === null ? 'Waiting for a fresh player count' : `Waiting for ${players} player(s) to leave`, automatic: true } }); db().prepare('UPDATE server_deployments SET players = ?, checked_at = ? WHERE id = ?').run(players, new Date().toISOString(), id); if (players !== 0) { deploymentStage(id, players === null ? 'Waiting for a fresh player count' : `Waiting for ${players} player(s) to leave`); return; } + } else if (action !== 'start') { + if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'waiting'").get(id)) return; + recordProgress(id, { phase: 'eligibility', blocker: null, checks: [{ key: 'stopped', label: 'Stopped container permits disruption', + source: 'komodo', result: 'pass', actual: health.state ?? null }] }); } // Requests can arrive during the probes. Recheck before claiming the action. const latestReason = blocker(server, id, action); - if (latestReason) { deploymentStage(id, latestReason); return; } + if (latestReason) { deploymentStage(id, latestReason.message, { phase: 'eligibility', blocker: latestReason }); return; } if (sourceEligibility(operation) !== 'ready') return; if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL AND state = 'waiting'").get(id)) return; if (action === 'rebuild') { await rebuildServer(id); return; } @@ -155,13 +193,26 @@ async function processOperation(operation: Operation): Promise { } return; } - db().prepare('UPDATE server_deployments SET submitted_at = ? WHERE id = ?').run(new Date().toISOString(), id); - deploymentStage(id, 'Stopping container'); + db().transaction(() => { + db().prepare('UPDATE server_deployments SET submitted_at = ? WHERE id = ?').run(new Date().toISOString(), id); + deploymentStage(id, 'Stopping container', { phase: 'container-action', blocker: null, checks: [], + checkpoint: { key: 'eligibility', label: 'Eligibility checks passed' }, command: { state: 'attempted', message: 'Stop submission attempted.' } }); + })(); // Persist intent first; a lost reply must not issue another stop later. try { const result = await komodoRuntimeAction(server, 'stop'); - if (!result.ok) { finishDeployment(id, { ok: false, message: result.message }); return; } - db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ?").run(id); - deploymentStage(id, 'Waiting for container to stop'); - } catch (err) { finishDeployment(id, { ok: false, message: (err as Error).message }); } + if (!result.ok) { + recordProgress(id, { command: { state: result.kind === 'no-name' || result.kind === 'unconfigured' ? 'rejected' : 'uncertain', message: result.message } }); + finishDeployment(id, { ok: false, message: result.message }); return; + } + db().transaction(() => { + db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ?").run(id); + deploymentStage(id, 'Waiting for container to stop', { phase: 'verify-stopped', + checkpoint: { key: 'container-action', label: 'Stop command accepted' }, + command: { state: 'accepted', message: 'Komodo accepted the stop request; stopped state still requires verification.' } }); + })(); + } catch (err) { + recordProgress(id, { command: { state: 'uncertain', message: (err as Error).message } }); + finishDeployment(id, { ok: false, message: (err as Error).message }); + } } diff --git a/src/lib/ui/components/LaunchRecipe.svelte b/src/lib/ui/components/LaunchRecipe.svelte index 2005a3c..4d6e012 100644 --- a/src/lib/ui/components/LaunchRecipe.svelte +++ b/src/lib/ui/components/LaunchRecipe.svelte @@ -1,11 +1,11 @@
-
{historical ? 'Recorded launch files' : 'Current promoted setup'}

{recipe.mapName}

{recipe.game === 't6' ? 'Black Ops II' : 'Black Ops I'}
+
{historical ? selectionStatus === 'queued' ? 'Queued setup capture' : selectionStatus === 'attempted' ? 'Selected for this attempt' : 'Recorded launch files' : 'Current promoted setup'}

{recipe.mapName}

{recipe.game === 't6' ? 'Black Ops II' : 'Black Ops I'}
Configuration
{recipe.configSource}
Container image
{recipe.image}
@@ -14,7 +14,7 @@
{recipe.files.length} managed file{recipe.files.length === 1 ? '' : 's'} {historical ? 'recorded for this deployment' : 'in the current setup'}
    {#each recipe.files as file}
  • {file.name}{file.version !== null ? `v${file.version}` : 'Generated'} · {file.kind} · {file.size.toLocaleString()} bytes
    {file.path}SHA-256 {file.sha256.slice(0, 12)}
    • {/each}
-

{historical ? 'Recorded' : 'Previewed'} {$timeFormat.dateTime(recipe.capturedAt)}. {historical ? 'This records the files selected for this deployment. Launch again uses the latest promoted setup for this map.' : 'This preview shows the promoted setup at the time above. The deployment history records the files selected when provisioning begins.'}

+

{historical ? 'Recorded' : 'Previewed'} {$timeFormat.dateTime(recipe.capturedAt)}. {historical ? selectionStatus === 'queued' ? 'Versions are selected again when creation begins. This queued capture does not pin them.' : selectionStatus === 'attempted' ? 'These files were selected for this attempt; this does not confirm installation succeeded. Launch again uses the latest promoted setup.' : 'This records the saved setup for this deployment. Launch again uses the latest promoted setup for this map.' : 'This preview shows the promoted setup at the time above. The deployment history records the files selected when provisioning begins.'}

Recipe {recipe.fingerprint.slice(0, 16)}
diff --git a/src/lib/ui/components/ProvisioningActivity.svelte b/src/lib/ui/components/ProvisioningActivity.svelte new file mode 100644 index 0000000..c007e4f --- /dev/null +++ b/src/lib/ui/components/ProvisioningActivity.svelte @@ -0,0 +1,132 @@ + + +
+
Activity

Recent updates

{detail.activity.total} event{detail.activity.total === 1 ? '' : 's'}
+

{open ? 'Recent updates stay in place while you read the full history.' : 'Newest first · updates automatically'}

+ {#if recent.length}
    + {#each recent as event (event.id)}

  1. {event.message}

    2. {/each} +
{:else}

No detailed activity was recorded. The recorded outcome and timestamps are preserved above.

{/if} + +
+ Full activity history {detail.activity.total} events +
+ {#if newEvents > 0}
{newEvents} new event{newEvents === 1 ? '' : 's'} since this view was loaded.
{/if} +
{ event.preventDefault(); void load(); }}> + + + +
+ {#if error}{/if} + {#if loaded}

{appliedOrder === 'newest' ? 'Newest first' : 'Oldest first'}{appliedQuery ? ` · Matching “${appliedQuery}”` : ' · All messages'}

{/if} + {#if loading && !loaded}

Loading activity…

+ {:else if loaded && !entries.length}

{appliedQuery ? 'No activity matches this search.' : 'No detailed activity recorded.'}

+ {:else}
    + {#each entries as event (event.id)}

  1. {event.message}

    2. {/each} +
{/if} + {#if loaded}
Showing {entries.length} of {total} matching events{#if cursor !== null}{/if}
{/if} +
+
+
+ + diff --git a/src/lib/ui/components/ProvisioningProgress.svelte b/src/lib/ui/components/ProvisioningProgress.svelte new file mode 100644 index 0000000..fbfa2d5 --- /dev/null +++ b/src/lib/ui/components/ProvisioningProgress.svelte @@ -0,0 +1,123 @@ + + +
+
Execution

Progress and evidence

+ {#if !progress} +

{detail.deployment.stage}

Detailed checkpoints were not recorded for this operation. Its recorded outcome and original activity remain available below.

+ {:else} +
    + {#each groups as group (group.key)} +
  1. +
    + { event.preventDefault(); expanded[group.key] = !(expanded[group.key] ?? ['current', 'waiting', 'failed'].includes(group.state)); }} + onkeydown={event => { + if (event.key !== 'Enter' && event.key !== ' ') return; + event.preventDefault(); + if (!event.repeat) expanded[group.key] = !(expanded[group.key] ?? ['current', 'waiting', 'failed'].includes(group.state)); + }} + > + + {group.label} + {progressStateLabels[group.state]} + +
      + {#each group.steps as step (step.key)} +
    1. + +
      {step.label}{progressStateLabels[step.state]}{#if step.at} · {/if}
      +
      2. + {/each} +
    +
    +
    2. + {/each} +
+ {#if progress.command} +
+

{progress.command.state === 'attempted' ? 'Command submission attempted' : progress.command.state === 'accepted' ? 'Remote request accepted' : progress.command.state === 'rejected' ? 'Remote request rejected' : 'Remote response uncertain'}

+

{progress.command.message}

+ +
+ {/if} + {#if progress.checkpoints.some(checkpoint => !groups.some(group => group.steps.some(step => step.key === checkpoint.key)))} +
Recorded checkpoints
    + {#each progress.checkpoints.filter(checkpoint => !groups.some(group => group.steps.some(step => step.key === checkpoint.key))) as checkpoint (checkpoint.key)} +
  • {checkpoint.label}
    • + {/each} +
+ {/if} +

Latest recorded checks

{#if progress.observedAt}Evaluated {/if}
+ {#if progress.checks.length} +
+ {#each progress.checks as check (check.key)} +
+
{check.label}{evidenceLabels[check.result]}
+ {#if check.expected !== null || check.actual !== null}
+ {#if check.expected !== null}
Expected
{check.expected}
{/if} +
Observed
{check.actual ?? 'Unknown'}
+
{/if} + {#if check.message}

{check.message}

{/if} +
{check.source ? sources[check.source] : 'Source not recorded'}Checked {#if check.receivedAt}Evidence received {/if}
+
+ {/each} +
+ {:else}

No check evidence has been recorded for this operation yet.

{/if} + {#if detail.deployment.expected_map}

Selected engine map: {detail.deployment.expected_map}. Engine-map verification does not independently verify a variant or start location.

{/if} + {/if} +
+ + diff --git a/src/lib/ui/provisioning-presentation.ts b/src/lib/ui/provisioning-presentation.ts new file mode 100644 index 0000000..5fbc843 --- /dev/null +++ b/src/lib/ui/provisioning-presentation.ts @@ -0,0 +1,202 @@ +import { provisioningPhases, type ProvisioningDetail, type ProvisioningEvent, type ProvisioningEventPage, type ProvisioningPhase } from '$lib/provisioning-detail.js'; +import { actionCompleted, actionLabels, provisioningActions, type ProvisioningAction } from '$lib/provisioning.js'; + +export const activeProvisioningStates = ['queued', 'waiting', 'running', 'awaiting-health']; +export type ProgressState = 'done' | 'current' | 'waiting' | 'failed' | 'pending' | 'cancelled' | 'unknown'; +export interface ProgressStep { key: ProvisioningPhase; label: string; state: ProgressState; at: string | null } +export interface ProgressGroup { key: string; label: string; steps: ProgressStep[]; state: ProgressState } + +const phaseLabels: Record = { + requested: 'Request recorded', eligibility: 'Check eligibility and capacity', 'startup-queue': 'Wait for a startup slot', + prepare: 'Prepare server files', 'create-deployment': 'Create Komodo deployment', 'container-action': 'Start container', + 'remove-container': 'Stop and remove container', 'delete-deployment': 'Delete Komodo deployment', cleanup: 'Remove files and registration', + 'verify-container': 'Confirm container readiness', 'verify-game': 'Verify game status', 'verify-map': 'Verify selected engine map', + 'verify-stopped': 'Confirm container is stopped' +}; +type GroupDefinition = { key: string; label: string; phases: ProvisioningPhase[] }; +const request: GroupDefinition = { key: 'request', label: 'Request and eligibility', phases: ['requested', 'eligibility'] }; +const creation: GroupDefinition = { key: 'create', label: 'Create and start', phases: ['startup-queue', 'prepare', 'create-deployment', 'container-action'] }; +const removal: GroupDefinition = { key: 'remove', label: 'Remove the instance', phases: ['remove-container', 'delete-deployment', 'cleanup'] }; +const readiness: GroupDefinition = { key: 'verify', label: 'Verify readiness', phases: ['verify-container', 'verify-game', 'verify-map'] }; +function definitions(action: ProvisioningAction): GroupDefinition[] { + if (action === 'create') return [{ ...request, phases: ['requested'] }, creation, readiness]; + if (action === 'rebuild') return [request, removal, { ...creation, label: 'Create a fresh instance' }, readiness]; + if (action === 'destroy') return [request, removal]; + return [request, { key: 'command', label: `${actionLabels[action]} the existing container`, phases: ['container-action'] }, + { ...readiness, phases: action === 'stop' ? ['verify-stopped'] : ['verify-container', 'verify-game'] }]; +} + +export function provisioningGroups(detail: ProvisioningDetail): ProgressGroup[] { + const progress = detail.progress; + if (!progress) return []; + const groups = definitions(detail.deployment.action); + const phases = groups.flatMap(group => group.phases); + const current = progress.phase ? phases.indexOf(progress.phase) : -1; + const active = activeProvisioningStates.includes(detail.deployment.state); + return groups.map(group => { + const steps = group.phases.map((key): ProgressStep => { + const checkpoint = progress.checkpoints.find(item => item.key === key); + const checkKey = key === 'verify-container' ? 'container' : key === 'verify-game' ? 'game' : key === 'verify-map' ? 'map' : key === 'verify-stopped' ? 'stopped' : null; + const check = checkKey ? progress.checks.find(item => item.key === checkKey) : null; + const index = phases.indexOf(key); + const isCurrent = key === progress.phase; + const state: ProgressState = check ? ({ pass: 'done', fail: 'failed', waiting: 'waiting', unknown: 'unknown' } as const)[check.result] + : checkpoint ? 'done' + : isCurrent && ['failed', 'interrupted'].includes(detail.deployment.state) ? 'failed' + : isCurrent && active ? progress.blocker || ['waiting', 'queued'].includes(detail.deployment.state) ? 'waiting' : 'current' + : current >= 0 && index >= current && detail.deployment.state === 'cancelled' ? 'cancelled' + : active && current >= 0 && index > current ? 'pending' : 'unknown'; + const label = key === 'container-action' && ['start', 'restart', 'stop'].includes(detail.deployment.action) + ? `${actionLabels[detail.deployment.action]} existing container` : phaseLabels[key]; + return { key, label, state, at: check?.checkedAt ?? checkpoint?.at ?? (isCurrent ? progress.phaseStartedAt : null) }; + }); + const states = steps.map(step => step.state); + const state = (['failed', 'current', 'waiting'] as const).find(value => states.includes(value)) + ?? (states.every(value => value === 'done') ? 'done' : states.includes('unknown') ? 'unknown' : states.includes('cancelled') ? 'cancelled' : 'pending'); + return { key: group.key, label: group.label, steps, state }; + }); +} + +export const progressStateLabels: Record = { + done: 'Completed', current: 'In progress', waiting: 'Waiting', failed: 'Needs attention', pending: 'Not started', cancelled: 'Not run', unknown: 'Not recorded' +}; +export const progressIcons: Record = { done: '✓', current: '•', waiting: '◷', failed: '!', pending: '○', cancelled: '–', unknown: '?' }; + +export function provisioningOutcome(detail: ProvisioningDetail) { + const { deployment: operation, progress } = detail; + const active = activeProvisioningStates.includes(operation.state); + const failure = ['failed', 'interrupted'].includes(operation.state); + const headline = operation.state === 'done' ? actionCompleted[operation.action] + : operation.state === 'cancelled' ? `${actionLabels[operation.action]} request cancelled` + : failure ? `${actionLabels[operation.action]} ${operation.state === 'interrupted' ? 'interrupted' : 'failed'}` + : operation.state === 'warning' ? `${actionLabels[operation.action]} completed with warnings` + : progress?.blocker?.message ?? operation.stage; + let description: string; + if (detail.canResumeCleanup) description = 'The remote deployment was deleted. Resume the remaining local cleanup without repeating container removal.'; + else if (detail.canResumeRebuild) description = 'The remote deployment was deleted. Resume local cleanup, then create a fresh instance in the reserved server slot.'; + else if (detail.canResumeVerification) description = 'Resume observation of the existing instance. This does not create, restart, or rebuild a container.'; + else if (operation.state === 'cancelled') description = 'This request is cancelled. Recorded prior work is preserved below; remaining work will not run.'; + else if (failure) description = progress?.command?.state === 'uncertain' + ? 'The remote response was uncertain. Inspect the recorded result and server before requesting another operation.' + : 'Review the recorded failure and execution evidence before submitting another operation.'; + else if (operation.state === 'warning') description = 'The operation finished. Review the recorded warnings below for any remaining issue.'; + else if (operation.state === 'done') description = !progress + ? 'This operation was recorded as completed. Detailed verification evidence was not recorded.' + : operation.action === 'destroy' ? 'The instance removal completed. This record preserves its results after deletion.' + : operation.action === 'stop' ? 'The container was confirmed stopped. Its files and registration were retained.' + : 'The operation completed its readiness checks. These recorded results describe that attempt; current health is available on the server page.'; + else if (active && progress?.phase === 'cleanup' && !progress.blocker) description = operation.action === 'rebuild' + ? 'Remote deployment deletion was confirmed. Local cleanup must finish before a fresh instance is created in the reserved slot.' + : 'Remote deployment deletion was confirmed. The remaining local files and registration cleanup will continue without repeating container removal.'; + else if (progress?.blocker) description = progress.blocker.automatic + ? 'The worker will recheck this condition and continue automatically when eligible.' + : 'Review the blocking condition below before this operation can continue.'; + else if (operation.state === 'queued') description = 'The request is saved. It will start when its dependencies and startup capacity allow.'; + else if (operation.state === 'waiting') description = 'Waiting for the worker’s eligibility checks. The recorded stage below describes the last known condition.'; + else description = operation.action === 'destroy' ? 'Removal is in progress. Confirmed checkpoints and remaining cleanup are recorded below.' + : operation.action === 'rebuild' ? 'The old instance is removed before a fresh instance is created in its reserved slot. Follow each checkpoint below.' + : 'The worker is executing this request and checking its resulting state.'; + return { active, headline, description }; +} + +/** Accept return context only from the local operation surfaces we expose. */ +export function provisioningReturn(value: string | null | undefined): { href: string; label: string } { + const fallback = { href: '/provisioning', label: 'Back to Provisioning' }; + if (!value || !value.startsWith('/') || value.startsWith('//') || value.includes('\\')) return fallback; + try { + const url = new URL(value, 'https://electric-cherry.invalid'); + if (url.origin !== 'https://electric-cherry.invalid') return fallback; + if (url.pathname === '/provisioning') { + const params = new URLSearchParams(); + for (const key of ['view', 'action', 'status', 'server', 'limit']) { + const item = url.searchParams.get(key); + if (item) params.set(key, item); + } + return { href: '/provisioning' + (params.size ? `?${params}` : ''), label: 'Back to Provisioning' }; + } + const match = url.pathname.match(/^\/(campaigns|betas|maintenance|collections\/launches)\/[^/]+$/); + if (match) return { href: url.pathname, label: `Back to ${match[1] === 'collections/launches' ? 'collection' : match[1] === 'campaigns' ? 'campaign' : match[1] === 'betas' ? 'beta' : 'maintenance'}` }; + } catch { /* Fall back to the local list. */ } + return fallback; +} + +export function elapsedTime(start: string | null | undefined, end: number): string { + if (!start || !Number.isFinite(Date.parse(start))) return 'Not recorded'; + const seconds = Math.max(0, Math.floor((end - Date.parse(start)) / 1000)); + if (seconds < 60) return `${seconds}s`; + if (seconds < 3600) return `${Math.floor(seconds / 60)}m ${seconds % 60}s`; + return `${Math.floor(seconds / 3600)}h ${Math.floor(seconds % 3600 / 60)}m`; +} + +const record = (value: unknown): value is Record => !!value && typeof value === 'object' && !Array.isArray(value); +const text = (value: unknown): value is string => typeof value === 'string'; +const number = (value: unknown): value is number => typeof value === 'number' && Number.isFinite(value); +const nullableText = (value: unknown) => value === null || text(value); +const timestamp = (value: unknown): value is string => text(value) && Number.isFinite(Date.parse(value)); +const nullableTime = (value: unknown) => value === null || timestamp(value); +const scalar = (value: unknown) => value === null || text(value) || number(value); +const localLink = (value: unknown) => value === null || (text(value) && value.startsWith('/') && !value.startsWith('//') && !value.includes('\\')); +const optionalLink = (value: unknown) => value === undefined || localLink(value); +const phase = (value: unknown) => value === null || provisioningPhases.includes(value as ProvisioningPhase); +const event = (value: unknown): value is ProvisioningEvent => record(value) && number(value.id) && timestamp(value.created_at) && text(value.message); + +export function isProvisioningEventPage(value: unknown): value is ProvisioningEventPage { + return record(value) && Array.isArray(value.events) && value.events.every(event) + && number(value.total) && number(value.totalEvents) && (value.latestId === null || number(value.latestId)) + && number(value.watermark) && (value.nextCursor === null || number(value.nextCursor)); +} + +/** Validate every nested shape consumed by the page before replacing its last good snapshot. */ +export function isProvisioningDetail(value: unknown, id: string): value is ProvisioningDetail { + if (!record(value) || !record(value.deployment)) return false; + const operation = value.deployment; + if (operation.id !== id || !provisioningActions.includes(operation.action as ProvisioningAction) + || !['server', 'state', 'stage', 'triggered_by', 'warnings'].every(key => text(operation[key])) + || !timestamp(operation.started_at) || !nullableTime(operation.finished_at) || !nullableTime(operation.server_deleted_at) + || !nullableText(operation.error) || !number(operation.server_exists) + || (operation.submitted_at !== undefined && !nullableTime(operation.submitted_at)) + || (operation.expected_map !== undefined && !nullableText(operation.expected_map))) return false; + if (!timestamp(value.generatedAt) || !Array.isArray(value.events) || !value.events.every(event) + || !Array.isArray(value.warningMessages) || !value.warningMessages.every(text) + || !['canResumeVerification', 'canResumeCleanup', 'canResumeRebuild', 'canCancelRemoval'].every(key => typeof value[key] === 'boolean') + || !nullableText(value.blueprintName)) return false; + if (!record(value.activity) || !number(value.activity.total) || !(value.activity.latestId === null || number(value.activity.latestId)) + || !(value.activity.failureExcerpt === null || event(value.activity.failureExcerpt))) return false; + if (!Array.isArray(value.sources) || !value.sources.every(source => record(source) + && ['campaign', 'beta', 'maintenance', 'watchdog'].includes(source.kind as string) + && text(source.source_id) && text(source.label) && localLink(source.href))) return false; + if (!(value.collection === null || record(value.collection) && text(value.collection.id) && text(value.collection.name))) return false; + if (!record(value.capacity) || !timestamp(value.capacity.evaluatedAt) || !Array.isArray(value.capacity.pools) + || !value.capacity.pools.every(pool => record(pool) && ['startup', 'restart'].includes(pool.kind as string) + && number(pool.limit) && number(pool.used) && typeof pool.ownsSlot === 'boolean' && (pool.requestOrder === null || number(pool.requestOrder)) + && Array.isArray(pool.holders) && pool.holders.every(holder => record(holder) && nullableText(holder.id) && text(holder.server)))) return false; + if (value.recipe !== null) { + const recipe = value.recipe; + if (!record(recipe) || !['t5', 't6'].includes(recipe.game as string) || !timestamp(recipe.capturedAt) + || !['mapSlug', 'mapName', 'expectedMap', 'image', 'configFile', 'configSource', 'fingerprint'].every(key => text(recipe[key])) + || !Array.isArray(recipe.files) || !recipe.files.every(file => record(file) + && ['name', 'kind', 'path', 'sha256'].every(key => text(file[key])) && number(file.size) && (file.version === null || number(file.version)))) return false; + } + if (value.progress !== null) { + const progress = value.progress; + if (!record(progress) || progress.version !== 1 || !phase(progress.phase) || !nullableTime(progress.phaseStartedAt) || !nullableTime(progress.observedAt) + || ![null, 'queued', 'attempted'].includes(progress.recipeStatus as null | string)) return false; + if (!(progress.blocker === null || record(progress.blocker) && text(progress.blocker.code) && text(progress.blocker.message) + && typeof progress.blocker.automatic === 'boolean' && optionalLink(progress.blocker.href) + && (progress.blocker.label === undefined || nullableText(progress.blocker.label)))) return false; + if (!Array.isArray(progress.checks) || !progress.checks.every(check => record(check) + && ['container', 'game', 'map', 'occupancy', 'stopped'].includes(check.key as string) + && ['pass', 'waiting', 'fail', 'unknown'].includes(check.result as string) + && [null, 'komodo', 'rcon', 'telemetry', 'combined'].includes(check.source as null | string) + && text(check.label) && timestamp(check.checkedAt) && nullableTime(check.receivedAt) + && scalar(check.expected) && scalar(check.actual) && nullableText(check.message))) return false; + if (!Array.isArray(progress.checkpoints) || !progress.checkpoints.every(checkpoint => record(checkpoint) + && text(checkpoint.key) && text(checkpoint.label) && timestamp(checkpoint.at) && phase(checkpoint.phase))) return false; + if (!(progress.command === null || record(progress.command) && ['attempted', 'accepted', 'rejected', 'uncertain'].includes(progress.command.state as string) + && timestamp(progress.command.at) && text(progress.command.message))) return false; + if (!(progress.recovery === null || record(progress.recovery) && ['held', 'released', 'transferred', 'unknown'].includes(progress.recovery.state as string) + && timestamp(progress.recovery.at) && text(progress.recovery.message) + && (progress.recovery.operationId === undefined || nullableText(progress.recovery.operationId)))) return false; + } + return true; +} diff --git a/src/lib/ui/provisioning-refresh.ts b/src/lib/ui/provisioning-refresh.ts new file mode 100644 index 0000000..81d3c93 --- /dev/null +++ b/src/lib/ui/provisioning-refresh.ts @@ -0,0 +1,131 @@ +export type ProvisioningMutation = 'resume' | 'cancel'; + +export interface ProvisioningRefreshState { + id: string; + detail: T | null; + refreshing: boolean; + busy: ProvisioningMutation | null; + loadError: string; + actionError: string; + notice: string; + refreshedAt: string | null; +} + +interface Options { + onChange: (state: ProvisioningRefreshState) => void; + validate: (value: unknown, id: string) => value is T; + fetcher?: typeof fetch; + now?: () => string; + timeoutMs?: number; +} + +/** Reads and action feedback belong to a particular visit, even when its ID is revisited. */ +export function createProvisioningRefresh(options: Options) { + const request = options.fetcher ?? fetch; + const now = options.now ?? (() => new Date().toISOString()); + let visit = 0; + let read = 0; + let disposed = false; + let pending: { controller: AbortController; promise: Promise; timer?: ReturnType } | null = null; + let mutation: AbortController | null = null; + let state: ProvisioningRefreshState = { + id: '', detail: null, refreshing: false, busy: null, loadError: '', actionError: '', notice: '', refreshedAt: null + }; + const publish = (patch: Partial>) => { + state = { ...state, ...patch }; + if (!disposed) options.onChange(state); + }; + function invalidateRead() { + read++; + if (pending) { + clearTimeout(pending.timer); + pending.controller.abort(); + pending = null; + } + } + + async function load(force = false, reconcile = false): Promise { + if (disposed || !state.id || (state.busy && !reconcile)) return; + if (pending && !force) return pending.promise; + invalidateRead(); + const currentVisit = visit; + const currentRead = read; + const id = state.id; + const current = () => !disposed && currentVisit === visit && currentRead === read && id === state.id; + const controller = new AbortController(); + const entry = { controller, promise: Promise.resolve(), timer: undefined as ReturnType | undefined }; + let timedOut = false; + pending = entry; + publish({ refreshing: true }); + entry.timer = setTimeout(() => { timedOut = true; controller.abort(); }, options.timeoutMs ?? 15_000); + entry.promise = (async () => { + try { + const response = await request(`/api/provisioning/${encodeURIComponent(id)}?events=recent`, { signal: controller.signal, cache: 'no-store' }); + if (!response.ok) throw new Error(response.status === 404 ? 'Provisioning operation not found.' : 'Could not refresh this operation.'); + const detail: unknown = await response.json(); + if (!options.validate(detail, id)) throw new Error('The operation response was incomplete. Try refreshing again.'); + if (current()) publish({ detail, loadError: '', refreshedAt: now() }); + } catch (error) { + if (current() && (!controller.signal.aborted || timedOut)) publish({ + loadError: timedOut ? 'Operation refresh timed out. Try again.' : error instanceof Error ? error.message : 'Could not refresh this operation.' + }); + } finally { + clearTimeout(entry.timer); + if (pending === entry) pending = null; + if (current()) publish({ refreshing: false }); + } + })(); + return entry.promise; + } + + function setOperation(id: string, detail: T | null = null, refreshedAt: string | null = null) { + if (disposed) return; + visit++; + invalidateRead(); + mutation?.abort(); + mutation = null; + publish({ id, detail, refreshedAt, refreshing: false, busy: null, loadError: '', actionError: '', notice: '' }); + if (!detail) void load(); + } + + async function act(action: ProvisioningMutation, successMessage: string): Promise { + if (disposed || !state.id || state.busy) return; + const currentVisit = visit; + const id = state.id; + const current = () => !disposed && currentVisit === visit && id === state.id; + invalidateRead(); + const controller = new AbortController(); + mutation = controller; + let timedOut = false; + const timer = setTimeout(() => { timedOut = true; controller.abort(); }, options.timeoutMs ?? 15_000); + publish({ busy: action, refreshing: false, actionError: '', notice: '' }); + try { + const response = await request(`/api/provisioning/${encodeURIComponent(id)}/${action}`, { method: 'POST', signal: controller.signal }); + const body = await response.json() as { error?: string }; + if (!response.ok) throw new Error(body.error ?? 'The operation changed before this request could be accepted.'); + if (current()) publish({ notice: successMessage }); + } catch (error) { + if (current() && (!controller.signal.aborted || timedOut)) publish({ actionError: timedOut + ? 'The request timed out and may have reached the server. Review the refreshed operation before trying again.' + : error instanceof Error ? error.message : 'Could not confirm the request. Review the operation before trying again.' }); + } finally { + clearTimeout(timer); + if (mutation === controller) mutation = null; + // A failed reconciliation is a read error, never a failed accepted mutation. + if (current()) { + await load(true, true); + if (current()) publish({ busy: null }); + } + } + } + + function dispose() { + disposed = true; + visit++; + invalidateRead(); + mutation?.abort(); + mutation = null; + } + + return { setOperation, refresh: () => load(), forceRefresh: () => load(true), act, dispose }; +} diff --git a/src/routes/api/deployments/[id]/+server.ts b/src/routes/api/deployments/[id]/+server.ts index d3134a4..772f996 100644 --- a/src/routes/api/deployments/[id]/+server.ts +++ b/src/routes/api/deployments/[id]/+server.ts @@ -2,8 +2,8 @@ import { json, error } from '@sveltejs/kit'; import { db } from '$lib/server/db/index.js'; import { getDeployment } from '$lib/server/provisioning/index.js'; -export async function GET({ params }: { params: { id: string } }) { - const lifecycle = getDeployment(params.id); +export async function GET({ params, url }: { params: { id: string }; url?: URL }) { + const lifecycle = getDeployment(params.id, { events: url?.searchParams.get('events') === 'recent' ? 'recent' : 'all' }); if (lifecycle) return json(lifecycle); const dep = db() .prepare('SELECT * FROM deployments WHERE id = ?') diff --git a/src/routes/api/provisioning/[id]/events/+server.ts b/src/routes/api/provisioning/[id]/events/+server.ts new file mode 100644 index 0000000..c044d4d --- /dev/null +++ b/src/routes/api/provisioning/[id]/events/+server.ts @@ -0,0 +1,24 @@ +import { json, error } from '@sveltejs/kit'; +import { db } from '$lib/server/db/index.js'; +import { deploymentActivity } from '$lib/server/provisioning/index.js'; + +function integer(params: URLSearchParams, key: string): number | undefined { + const raw = params.get(key); + if (raw === null) return undefined; + if (!/^\d+$/.test(raw)) error(400, `${key} must be a nonnegative integer.`); + const value = Number(raw); + if (!Number.isSafeInteger(value)) error(400, `${key} must be a nonnegative integer.`); + return value; +} + +export function GET({ params, url }: { params: { id: string }; url: URL }) { + if (!db().prepare('SELECT 1 FROM server_deployments WHERE id = ?').get(params.id)) error(404, 'Provisioning operation not found.'); + const order = url.searchParams.get('order') ?? 'newest'; + if (order !== 'newest' && order !== 'oldest') error(400, 'Choose newest or oldest activity order.'); + const query = url.searchParams.get('query') ?? ''; + if (query.length > 500) error(400, 'Activity search must be 500 characters or fewer.'); + const limit = integer(url.searchParams, 'limit'); + if (limit === 0) error(400, 'Activity limit must be at least 1.'); + return json(deploymentActivity(params.id, { order, query, limit, + cursor: integer(url.searchParams, 'cursor'), watermark: integer(url.searchParams, 'watermark') })); +} diff --git a/src/routes/provisioning/+page.svelte b/src/routes/provisioning/+page.svelte index 4eaf656..8461bff 100644 --- a/src/routes/provisioning/+page.svelte +++ b/src/routes/provisioning/+page.svelte @@ -128,7 +128,7 @@ {#snippet row(d: Deployment)} -
+
{actionLabels[d.action]} diff --git a/src/routes/provisioning/[id]/+page.server.ts b/src/routes/provisioning/[id]/+page.server.ts index 81b51b5..ad2dbbf 100644 --- a/src/routes/provisioning/[id]/+page.server.ts +++ b/src/routes/provisioning/[id]/+page.server.ts @@ -2,8 +2,8 @@ import { redirect, error } from '@sveltejs/kit'; import { getDeployment } from '$lib/server/provisioning/index.js'; import { db } from '$lib/server/db/index.js'; export function load({ params }: { params: { id: string } }) { - const detail = getDeployment(params.id); - if (detail) return {}; + const detail = getDeployment(params.id, { events: 'recent' }); + if (detail) return { detail }; if (db().prepare('SELECT id FROM deployments WHERE id = ?').get(params.id)) { redirect(307, '/campaigns/' + encodeURIComponent(params.id)); } diff --git a/src/routes/provisioning/[id]/+page.svelte b/src/routes/provisioning/[id]/+page.svelte index ed3c22b..46adb81 100644 --- a/src/routes/provisioning/[id]/+page.svelte +++ b/src/routes/provisioning/[id]/+page.svelte @@ -1,177 +1,178 @@ -{deployment ? actionLabels[deployment.action] + ' ' + deployment.server : 'Provisioning'} | Electric Cherry -
- - {#snippet crumbs()}{/snippet} - - {#if error}{/if} - {#if detail?.collection}← Back to {detail.collection.name} collection{/if} - {#if deployment} -
-
- {#if deployment.server_deleted_at}Deleted{:else if deployment.server_exists}View server{/if} -
-

{deployment.state === 'done' ? actionCompleted[deployment.action] : deployment.stage}

-

{explanation}

-
Total time {duration(deployment.started_at, deployment.finished_at ? new Date(deployment.finished_at).getTime() : now)}{#if active && stageStarted}This step {duration(stageStarted, now)}{/if}
- {#if detail?.canResumeVerification && !deployment.server_deleted_at}
{/if} - {#if detail?.canResumeRebuild && !deployment.server_deleted_at}
{/if} - {#if detail?.canResumeCleanup && !deployment.server_deleted_at}
{/if} - {#if detail?.canCancelRemoval && !deployment.server_deleted_at}

Available until removal begins. Cancelling keeps the server and its files.

{/if} - {#if deployment.server_exists && !deployment.server_deleted_at && deployment.state !== 'done' && deployment.state !== 'cancelled'}
Open startup logsDiagnose connection{#if deployment.stage.includes('map')}Review map configuration{/if}
{/if} - {#if deployment.error}{/if} - {#each warnings as warning}

{warning}

{/each} - {#if deployment.expected_map}

Expected map: {deployment.expected_map}

{/if} -
Started
{fmt(deployment.started_at)}
Finished
{deployment.finished_at ? fmt(deployment.finished_at) : 'In progress'}
Requested by
{deployment.triggered_by}
- {#if detail?.sources.length}
{#each detail.sources as source}{source.kind === 'campaign' ? 'Campaign' : source.kind === 'beta' ? 'Beta' : source.kind === 'watchdog' ? 'Maintenance finding' : 'Maintenance'} {source.source_id}{/each}
{/if} -
- {#if detail?.recipe}

{detail.blueprintName ? `Launched from ${detail.blueprintName}` : 'Recorded launch files'}

Launch this map with the latest promoted files using a fresh server ID and port. This does not repair the existing server.

{/if} - ({label,state:deployment.state === 'cancelled' ? 'pending' : index < currentStep ? 'done' : index === currentStep ? (!active && ['failed','interrupted'].includes(deployment.state) ? 'failed' : 'current') : 'pending'}))} events={detail?.events ?? []} {active} /> +{deployment ? `${actionLabels[deployment.action]} ${deployment.server}` : 'Provisioning'} | Electric Cherry - {:else if !error}

Loading provisioning operation...

{/if} +
+ + {#snippet crumbs()}{/snippet} + {#snippet actions()}{#if serverHref}{/if}{/snippet} + +
+ {#if deployment}
Operation
{/if} +
← {returnTo.label}
+
+ + {#if view.loadError}{/if} + + {#if deployment && detail && outcome} +
+
{#if deployment.server_deleted_at}Deleted identity{/if}{#if detail.progress?.recovery?.state === 'held'}Recovery capacity held{/if}
Operation outcome
+

{outcome.headline}

{outcome.description}

+ {#if deployment.error}{/if} + {#if detail.activity.failureExcerpt && detail.activity.failureExcerpt.message !== deployment.error && ['failed', 'interrupted'].includes(deployment.state)}

Last failure event · {fmt(detail.activity.failureExcerpt.created_at)}{detail.activity.failureExcerpt.message}

{/if} + {#if detail.warningMessages.length}

Recorded warnings

    {#each detail.warningMessages as warning}
  • {warning}
    • {/each}
{/if} + + {#if detail.canResumeVerification || detail.canResumeCleanup || detail.canResumeRebuild || detail.canCancelRemoval} +
+ {#if detail.canResumeVerification}{/if} + {#if detail.canResumeRebuild}{/if} + {#if detail.canResumeCleanup}{/if} + {#if detail.canCancelRemoval}

Available until removal begins. Cancelling keeps the server and its files.

{/if} +
+ {/if} + {#if view.actionError}{/if} + {#if view.notice}

{view.notice}

{/if} + {#if detail.progress?.blocker?.href}{detail.progress.blocker.label ?? 'View blocking work'} →{/if} + {#if detail.progress?.recovery}
{detail.progress.recovery.state === 'held' ? 'Recovery remains in progress' : detail.progress.recovery.state === 'released' ? 'Recovery capacity released' : detail.progress.recovery.state === 'transferred' ? 'Recovery transferred' : 'Recovery state not confirmed'}

{detail.progress.recovery.message}

Recorded {fmt(detail.progress.recovery.at)}{#if detail.progress.recovery.operationId && detail.progress.recovery.operationId !== deployment.id}View recovery operation →{/if}
{/if} + {#if serverHref && !['done', 'cancelled'].includes(deployment.state)}
Open server logsDiagnose connection{#if detail.progress?.phase === 'verify-map' || detail.progress?.checks.some(check => check.key === 'map' && check.result !== 'pass')}Review map configuration{/if}
{/if} +
+
{outcome.active ? 'Total elapsed' : 'Recorded duration'}{elapsedTime(deployment.started_at, deployment.finished_at ? Date.parse(deployment.finished_at) : now)}
+ {#if outcome.active}
Time in phase{elapsedTime(detail.progress?.phaseStartedAt, now)}
{/if} +
Last evaluated
+
+
Page updated · updates automatically
+
+ + {#key deployment.id} +
+
+ +
+ {#if detail.recipe}{/if} + {/key} + {:else}
Loading provisioning operation…
{/if}
diff --git a/tests/unit/create-server.test.ts b/tests/unit/create-server.test.ts index 9938de9..217d921 100644 --- a/tests/unit/create-server.test.ts +++ b/tests/unit/create-server.test.ts @@ -502,7 +502,10 @@ describe('server creation map selection', () => { expect(komodoDestroyDeployment).not.toHaveBeenCalled(); const detail = await (await GET({ params: { id: body.deploymentId } })).json(); expect(detail.deployment).toMatchObject({ server: serverName, action: 'destroy', state: 'waiting' }); - expect(detail.events).toHaveLength(2); + expect(detail.events.map((event: { message: string }) => event.message)).toEqual(expect.arrayContaining([ + 'Server destruction requested', 'Waiting to verify the server is empty' + ])); + expect(detail.events.every((event: { id: number }, index: number) => index === 0 || detail.events[index - 1].id < event.id)).toBe(true); const missing = await DELETE({ params: { name: 'missing-server' } }); expect(missing.status).toBe(404); }); diff --git a/tests/unit/provisioning-detail.test.ts b/tests/unit/provisioning-detail.test.ts new file mode 100644 index 0000000..0077a87 --- /dev/null +++ b/tests/unit/provisioning-detail.test.ts @@ -0,0 +1,204 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { newFixture, type Fixture } from '../helpers/fixture.js'; + +let fixture: Fixture; +beforeEach(async () => { vi.resetModules(); fixture = await newFixture(); }); +afterEach(async () => { await fixture.cleanup(); vi.restoreAllMocks(); }); + +describe('provisioning detail projection', () => { + it('records identical structured and legacy stage text once while preserving changed observations', async () => { + const { beginDeployment, deploymentStage, getDeployment } = await import('../../src/lib/server/provisioning/history.js'); + const id = beginDeployment('server', 'restart'); + const update = { phase: 'eligibility' as const, + blocker: { code: 'players', message: 'Waiting for 3 players', automatic: true } }; + deploymentStage(id, update.blocker.message, update); + deploymentStage(id, update.blocker.message, update); + expect(getDeployment(id)!.events.filter(event => event.message === update.blocker.message)).toHaveLength(1); + deploymentStage(id, 'Waiting for 2 players', { ...update, blocker: { ...update.blocker, message: 'Waiting for 2 players' } }); + const events = getDeployment(id)!.events; + expect(events.filter(event => event.message === 'Waiting for 2 players')).toHaveLength(1); + expect(events.filter(event => event.kind === 'blocker')).toHaveLength(1); + }); + + it('keeps legacy full activity while the page and opt-in API return a bounded snapshot without probing', async () => { + const history = await import('../../src/lib/server/provisioning/history.js'); + const komodo = await import('../../src/lib/server/integrations/komodo.js'); + const rcon = await import('../../src/lib/server/integrations/rcon.js'); + const inspect = vi.spyOn(komodo, 'komodoInspectHealthy'); + const status = vi.spyOn(rcon, 'rconStatusResult'); + const id = history.beginDeployment('t5_zombie_theater_1', 'create'); + for (let index = 0; index < 40; index++) history.recordDeploymentEvent(id, `Recorded activity ${index}`); + history.finishDeployment(id, { ok: false, message: 'A recorded startup failure' }); + for (let index = 0; index < 8; index++) history.recordDeploymentEvent(id, `Later observation ${index}`); + + const full = history.getDeployment(id)!; + const recent = history.getDeployment(id, { events: 'recent' })!; + expect(full.events).toHaveLength(50); + expect(recent.events).toEqual(full.events.slice(-5)); + expect(recent.activity).toMatchObject({ total: 50, latestId: full.events.at(-1)!.id, + failureExcerpt: { message: 'Failed: A recorded startup failure', kind: 'outcome' } }); + expect(recent.progress?.phase).toBe('requested'); + expect(recent.progress?.checkpoints.map(point => point.key)).toEqual(['requested']); + const { GET } = await import('../../src/routes/api/provisioning/[id]/+server.js'); + expect((await (await GET({ params: { id } })).json()).events).toHaveLength(50); + expect((await (await GET({ params: { id }, url: new URL('http://localhost/?events=recent') })).json()).events).toHaveLength(5); + const { load } = await import('../../src/routes/provisioning/[id]/+page.server.js'); + expect(load({ params: { id } })?.detail.events).toHaveLength(5); + expect(inspect).not.toHaveBeenCalled(); + expect(status).not.toHaveBeenCalled(); + }); + + it('retains missing sources, normalizes malformed warnings, and leaves legacy evidence unrecorded', async () => { + const { db } = await import('../../src/lib/server/db/index.js'); + const { getDeployment } = await import('../../src/lib/server/provisioning/history.js'); + db().prepare(`INSERT INTO server_deployments (id,server,action,state,stage,started_at,triggered_by,warnings) + VALUES ('legacy','old-server','restart','done','Old custom stage','2026-01-01','ui',?)`).run('["Check files",7,null,{"bad":true}]'); + db().prepare("INSERT INTO provisioning_sources VALUES ('legacy','campaign','missing','old-server')").run(); + db().prepare("INSERT INTO provisioning_sources VALUES ('legacy','watchdog','incident','old-server')").run(); + db().prepare(`INSERT INTO watchdog_incidents (id,server,kind,state,summary,first_seen_at,last_seen_at) + VALUES ('incident','old-server','health','resolved','Recovered game service','2026-01-01','2026-01-01')`).run(); + const detail = getDeployment('legacy')!; + expect(detail.progress).toBeNull(); + expect(detail.events).toEqual([]); + expect(detail.activity).toEqual({ total: 0, latestId: null, failureExcerpt: null }); + expect(detail.warningMessages).toEqual(['Check files', 'Some recorded warnings could not be read.']); + expect(detail.sources).toEqual([ + { kind: 'campaign', source_id: 'missing', server: 'old-server', label: 'Campaign missing', href: null }, + { kind: 'watchdog', source_id: 'incident', server: 'old-server', label: 'Recovered game service', href: '/maintenance/incident' } + ]); + db().prepare("UPDATE server_deployments SET warnings = 'broken json' WHERE id = 'legacy'").run(); + expect(getDeployment('legacy')!.warningMessages).toEqual(['Recorded warnings could not be read.']); + }); + + it('correlates reservations by operation ID without assigning a later same-name slot to historical work', async () => { + const { db } = await import('../../src/lib/server/db/index.js'); + const { beginDeployment, getDeployment, finishDeployment } = await import('../../src/lib/server/provisioning/history.js'); + db().prepare("INSERT INTO servers (name,komodo_name) VALUES ('server','server')").run(); + const old = beginDeployment('server', 'restart'); + finishDeployment(old, { ok: false, message: 'Timeout' }); + const current = beginDeployment('server', 'restart'); + db().prepare("INSERT INTO restart_rollout (server,requested_at,operation_id) VALUES ('server','2026-01-01',?)").run(current); + expect(getDeployment(old)!.capacity.pools[0]).toMatchObject({ kind: 'restart', used: 1, ownsSlot: false, + holders: [{ id: current, server: 'server' }] }); + expect(getDeployment(old)!.progress!.recovery).toBeNull(); + expect(getDeployment(current)!.capacity.pools[0].ownsSlot).toBe(true); + db().prepare('UPDATE restart_rollout SET healthy_at = ?').run(new Date().toISOString()); + expect(getDeployment(old)!.capacity.pools[0].used).toBe(0); + expect(getDeployment(old)!.deployment.state).toBe('failed'); + }); + + it('labels submission order independently of startup reservations and retains queued recipe status', async () => { + const { db } = await import('../../src/lib/server/db/index.js'); + const { beginDeployment, getDeployment } = await import('../../src/lib/server/provisioning/history.js'); + const { recordProgress } = await import('../../src/lib/server/provisioning/progress.js'); + const first = beginDeployment('first', 'create'); + const next = beginDeployment('next', 'create'); + db().prepare("UPDATE server_deployments SET state = 'queued' WHERE id = ?").run(next); + recordProgress(next, { phase: 'startup-queue', recipeStatus: 'queued', observedAt: null }); + const detail = getDeployment(next)!; + expect(detail.capacity.pools[0]).toMatchObject({ kind: 'startup', used: 1, requestOrder: 1, ownsSlot: false, + holders: [{ id: first, server: 'first' }] }); + expect(detail.progress).toMatchObject({ recipeStatus: 'queued', observedAt: null }); + }); + + it('preserves a failure outcome and evidence before an observation-only same-ID resume', async () => { + const { db } = await import('../../src/lib/server/db/index.js'); + const history = await import('../../src/lib/server/provisioning/history.js'); + db().prepare("INSERT INTO servers (name,komodo_name,game_port) VALUES ('server','server',28960)").run(); + const id = history.beginDeployment('server', 'create'); + db().prepare("INSERT INTO server_creation_jobs VALUES (?, 'server',28960,'zombie','theater','zombie_theater')").run(id); + history.deploymentStage(id, 'Waiting for game evidence', { phase: 'verify-game', checks: [ + { key: 'game', label: 'Game status', result: 'waiting', source: null } + ] }); + history.finishDeployment(id, { ok: false, message: 'Observation interrupted' }); + const failed = history.getDeployment(id)!; + expect(failed.progress?.phase).toBe('verify-game'); + expect(history.resumeVerification(id)).toBe(true); + const resumed = history.getDeployment(id)!; + expect(resumed.deployment).toMatchObject({ id, state: 'awaiting-health', finished_at: null, error: null }); + expect(resumed.progress).toMatchObject({ phase: 'verify-container', checks: [], observedAt: null }); + const outcomes = resumed.events.filter(event => event.kind === 'outcome'); + expect(outcomes).toHaveLength(1); + expect(JSON.parse(outcomes[0].details_json!)).toMatchObject({ state: 'failed', finishedAt: failed.deployment.finished_at, + error: 'Observation interrupted' }); + expect(resumed.events.some(event => event.kind === 'resumed')).toBe(true); + }); + + it('preserves a legacy cleanup failure and keeps deleted cleanup capability authoritative', async () => { + const { db } = await import('../../src/lib/server/db/index.js'); + const { getDeployment, resumeCleanup, REMOVING_SERVER } = await import('../../src/lib/server/provisioning/history.js'); + db().prepare(`INSERT INTO server_deployments (id,server,server_deleted_at,action,state,stage,started_at,finished_at,triggered_by,error) + VALUES ('cleanup','old-server','2026-01-02','destroy','interrupted',?,'2026-01-01','2026-01-02','ui','App restarted')`).run(REMOVING_SERVER); + expect(getDeployment('cleanup')).toMatchObject({ canResumeCleanup: true, deployment: { server_exists: 0 } }); + expect(resumeCleanup('cleanup')).toBe(true); + const result = getDeployment('cleanup')!; + expect(result.deployment).toMatchObject({ state: 'waiting', server_deleted_at: '2026-01-02' }); + const outcome = result.events.find(event => event.kind === 'outcome')!; + expect(JSON.parse(outcome.details_json!)).toMatchObject({ state: 'interrupted', finishedAt: '2026-01-02', error: 'App restarted' }); + expect(result.progress?.phase).toBe('cleanup'); + }); +}); + +describe('provisioning activity pagination', () => { + it('keeps polling payloads bounded for a retained history with thousands of events', async () => { + const { db } = await import('../../src/lib/server/db/index.js'); + const { beginDeployment, getDeployment } = await import('../../src/lib/server/provisioning/history.js'); + const id = beginDeployment('server', 'restart'); + const insert = db().prepare('INSERT INTO server_deployment_events (deployment_id,created_at,message) VALUES (?,?,?)'); + db().transaction(() => { + for (let index = 0; index < 5000; index++) insert.run(id, '2026-09-28T12:00:00Z', `Observation ${index} ${'x'.repeat(250)}`); + })(); + const full = getDeployment(id)!; + const recent = getDeployment(id, { events: 'recent' })!; + const fullBytes = Buffer.byteLength(JSON.stringify(full)); + const recentBytes = Buffer.byteLength(JSON.stringify(recent)); + expect(recent.activity.total).toBe(5001); + expect(recent.events).toHaveLength(5); + expect(recentBytes).toBeLessThan(fullBytes / 100); + if (process.env.PROVISIONING_BENCHMARK) { + const start = performance.now(); + for (let index = 0; index < 30; index++) getDeployment(id, { events: 'recent' }); + console.info({ events: full.events.length, fullBytes, recentBytes, meanRecentReadMs: (performance.now() - start) / 30 }); + } + }); + + it.each(['newest', 'oldest'] as const)('keeps stable %s pages under concurrent appends and searches literal text', async order => { + const { beginDeployment, recordDeploymentEvent } = await import('../../src/lib/server/provisioning/history.js'); + const { deploymentActivity } = await import('../../src/lib/server/provisioning/activity.js'); + const id = beginDeployment('server', 'stop'); + const other = beginDeployment('other', 'stop'); + for (let index = 0; index < 67; index++) { + recordDeploymentEvent(id, index === 30 ? 'literal %_needle' : `Event ${index}`); + recordDeploymentEvent(other, 'Unrelated event'); + } + let page = deploymentActivity(id, { order }); + const watermark = page.watermark; + const seen = [...page.events]; + expect(page.events).toHaveLength(25); + recordDeploymentEvent(id, 'New event during reading'); + while (page.nextCursor !== null) { + page = deploymentActivity(id, { order, watermark, cursor: page.nextCursor }); + seen.push(...page.events); + } + expect(seen).toHaveLength(68); + expect(new Set(seen.map(event => event.id)).size).toBe(68); + expect(seen.every(event => event.id <= watermark)).toBe(true); + expect(page.total).toBe(68); + expect(page.totalEvents).toBe(69); + expect(deploymentActivity(id, { query: '%_NEEDLE' }).events.map(event => event.message)).toEqual(['literal %_needle']); + expect(deploymentActivity(id, { query: 'not found' }).total).toBe(0); + }); + + it('validates cursor/search requests and caps page size without changing default legacy responses', async () => { + const { beginDeployment, recordDeploymentEvent } = await import('../../src/lib/server/provisioning/history.js'); + const { GET } = await import('../../src/routes/api/provisioning/[id]/events/+server.js'); + const id = beginDeployment('server', 'stop'); + for (let index = 0; index < 130; index++) recordDeploymentEvent(id, `Event ${index}`); + const response = GET({ params: { id }, url: new URL('http://localhost/?limit=999') }); + expect((await response.json()).events).toHaveLength(100); + for (const query of ['cursor=-1', 'watermark=NaN', 'limit=0', 'order=invalid', 'cursor=9007199254740992']) { + expect(() => GET({ params: { id }, url: new URL(`http://localhost/?${query}`) })).toThrow(); + } + expect(() => GET({ params: { id: 'missing' }, url: new URL('http://localhost/') })).toThrow(); + }); +}); diff --git a/tests/unit/provisioning-evidence.test.ts b/tests/unit/provisioning-evidence.test.ts new file mode 100644 index 0000000..079957b --- /dev/null +++ b/tests/unit/provisioning-evidence.test.ts @@ -0,0 +1,284 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { join } from 'node:path'; +import { newFixture, type Fixture } from '../helpers/fixture.js'; +import { gameEvent } from '../helpers/telemetry.js'; + +vi.mock('../../src/lib/server/integrations/komodo.js', () => ({ + komodoRestart: vi.fn(), komodoRuntimeAction: vi.fn(), komodoInspectHealthy: vi.fn(), + komodoCreateDeployment: vi.fn(), komodoDeploy: vi.fn(), komodoDestroyDeployment: vi.fn(), komodoDeleteDeployment: vi.fn(), + komodoEnsureTag: vi.fn(), komodoUpdateTagsOnResource: vi.fn() +})); +vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconGetPlayerCount: vi.fn(), rconStatusResult: vi.fn() })); +vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}) })); + +let fixture: Fixture; +const server = 't5_zombie_theater_1'; +const status = (map = 'zombie_theater') => ({ ok: true as const, endpoint: { host: 'test', port: 28960 }, rawReply: '', + status: { players: 0, map, gametype: 'zom', roster: [] } }); + +beforeEach(async () => { + vi.resetModules(); vi.resetAllMocks(); + vi.useFakeTimers({ toFake: ['Date'] }); + vi.setSystemTime(new Date('2026-09-28T12:00:00Z')); + fixture = await newFixture([server]); + for (const [key, value] of Object.entries({ KOMODO_ADDRESS: 'http://komodo.test', KOMODO_API_KEY: 'test', KOMODO_API_SECRET: 'test', + KOMODO_TARGET_SERVER: 'host', RCON_PASSWORD: 'test', DEPLOYMENT_CREATE_CONCURRENCY: '1', DEPLOYMENT_RESTART_CONCURRENCY: '1', + SERVER_PORT_MIN: '28960', SERVER_PORT_MAX: '28970' })) vi.stubEnv(key, value); + const komodo = await import('../../src/lib/server/integrations/komodo.js'); + for (const name of ['komodoRestart', 'komodoRuntimeAction', 'komodoCreateDeployment', 'komodoDeploy', 'komodoDestroyDeployment', 'komodoDeleteDeployment'] as const) { + vi.mocked(komodo[name]).mockResolvedValue({ ok: true }); + } + vi.mocked(komodo.komodoInspectHealthy).mockResolvedValue({ ok: true, state: 'running' }); + vi.mocked(komodo.komodoEnsureTag).mockImplementation(async name => ({ ok: true, id: name })); + vi.mocked(komodo.komodoUpdateTagsOnResource).mockResolvedValue({ ok: true }); + const rcon = await import('../../src/lib/server/integrations/rcon.js'); + vi.mocked(rcon.rconGetPlayerCount).mockResolvedValue(0); + vi.mocked(rcon.rconStatusResult).mockResolvedValue(status()); +}); +afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); vi.unstubAllEnvs(); }); + +async function register() { + const { db } = await import('../../src/lib/server/db/index.js'); + db().prepare(`INSERT INTO servers (name,komodo_type,komodo_name,game,map,game_mode,map_slug,game_port,host_dir) + VALUES (?, 'deployment', ?, 't5','zombie_theater','zombie','theater',28960,?)`).run(server, server, join(fixture.t5, server)); + const { saveServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); + saveServerKey(server, 'saved-key'); +} +async function detail(id: string) { + const { getDeployment } = await import('../../src/lib/server/provisioning/history.js'); + return getDeployment(id)!; +} +async function telemetry(players: number, map = 'zombie_theater') { + const { db } = await import('../../src/lib/server/db/index.js'); + const receivedAt = new Date().toISOString(); + db().prepare(`INSERT INTO server_telemetry (server,round,players,received_at,payload) VALUES (?,1,?,?,?) + ON CONFLICT(server) DO UPDATE SET players=excluded.players,received_at=excluded.received_at,payload=excluded.payload`) + .run(server, players, receivedAt, JSON.stringify({ ...gameEvent('heartbeat', 1, players, server, map), receivedAt })); + return receivedAt; +} + +describe('provisioning decision evidence', () => { + it('distinguishes queued recipes, attempted files, transient map checks, and telemetry completion without replaying startup', async () => { + const { queueCreateServer, processCreations } = await import('../../src/lib/server/provisioning/instances.js'); + const { komodoCreateDeployment, komodoDeploy, komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js'); + const { rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js'); + const job = queueCreateServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'saved-key' }); + if (!job.ok) throw new Error(job.message); + expect((await detail(job.deploymentId)).progress).toMatchObject({ phase: 'startup-queue', recipeStatus: 'queued', observedAt: null }); + + const { createScript, getLatestVersion } = await import('../../src/lib/server/fleet/scripts.js'); + const { db } = await import('../../src/lib/server/db/index.js'); + const { script } = createScript({ name: 'New promotion', targetRelPath: 'promoted.gsc', scopeType: 'game', scopeValue: 't5', content: Buffer.from('promoted while queued') }); + db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(getLatestVersion(script.id)!.id, script.id); + await processCreations(); + expect((await detail(job.deploymentId)).progress?.recipeStatus).toBe('attempted'); + expect((await detail(job.deploymentId)).recipe?.files.some(file => file.path === 'promoted.gsc')).toBe(true); + + vi.mocked(rconStatusResult).mockResolvedValueOnce(status('zombie_moon')); + vi.setSystemTime(Date.now() + 1000); + await processCreations(); + expect((await detail(job.deploymentId)).progress).toMatchObject({ phase: 'verify-map', blocker: { code: 'map-mismatch' }, + checks: [{ key: 'container', result: 'pass' }, { key: 'game', result: 'pass' }, { key: 'map', result: 'waiting', expected: 'zombie_theater', actual: 'zombie_moon' }] }); + vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: false, reason: 'container recovering' }); + await processCreations(); + expect((await detail(job.deploymentId)).progress?.checks).toEqual([expect.objectContaining({ key: 'container', result: 'waiting' })]); + + vi.setSystemTime(Date.now() + 1000); + const receivedAt = await telemetry(0); + await processCreations(); + const completed = await detail(job.deploymentId); + expect(completed.deployment.state).toBe('done'); + expect(completed.progress?.checks).toContainEqual(expect.objectContaining({ key: 'game', source: 'telemetry', result: 'pass', receivedAt })); + expect(completed.progress?.checkpoints).toContainEqual(expect.objectContaining({ key: 'verify-map' })); + expect(komodoCreateDeployment).toHaveBeenCalledTimes(1); + expect(komodoDeploy).toHaveBeenCalledTimes(1); + }); + + it('keeps an uncertain failed request immutable while observing and releasing its exact recovery slot', async () => { + await register(); + const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js'); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js'); + vi.mocked(komodoRestart).mockResolvedValueOnce({ ok: false, kind: 'unreachable', message: 'response lost' }); + const job = requestLifecycle(server, 'restart'); + if (!job.ok) throw new Error(job.error); + await processProvisioning(); + const failed = await detail(job.deploymentId); + expect(failed.progress).toMatchObject({ command: { state: 'uncertain' }, recovery: { state: 'held', operationId: job.deploymentId } }); + const failureEvents = failed.events.filter(event => event.kind === 'outcome'); + vi.setSystemTime(Date.now() + 15_000); + await telemetry(0); + await processProvisioning(); + const recovered = await detail(job.deploymentId); + expect(recovered.deployment).toMatchObject({ state: 'failed', finished_at: failed.deployment.finished_at }); + expect(recovered.progress).toMatchObject({ command: { state: 'uncertain' }, recovery: { state: 'released', operationId: job.deploymentId } }); + expect(recovered.events.filter(event => event.kind === 'outcome')).toEqual(failureEvents); + expect(komodoRestart).toHaveBeenCalledTimes(1); + }); + + it('records occupied and unknown evidence without treating either as zero or resetting eligibility timing', async () => { + await register(); + const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js'); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js'); + const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js'); + const job = requestLifecycle(server, 'restart'); + if (!job.ok) throw new Error(job.error); + vi.mocked(rconGetPlayerCount).mockResolvedValueOnce(null); + await processProvisioning(); + const unknown = await detail(job.deploymentId); + expect(unknown.progress?.checks).toEqual([expect.objectContaining({ key: 'occupancy', actual: null, source: null, result: 'unknown' })]); + vi.setSystemTime(Date.now() + 1000); + await telemetry(3); + await processProvisioning(); + expect((await detail(job.deploymentId)).progress).toMatchObject({ phaseStartedAt: unknown.progress?.phaseStartedAt, + blocker: { code: 'players-present' }, checks: [{ key: 'occupancy', actual: 3, source: 'telemetry' }] }); + vi.setSystemTime(Date.now() + 1000); + await telemetry(2); + await processProvisioning(); + expect((await detail(job.deploymentId)).progress?.phaseStartedAt).toBe(unknown.progress?.phaseStartedAt); + expect(komodoRestart).not.toHaveBeenCalled(); + }); + + it('preserves failed cleanup evidence through same-ID resume without another destructive command', async () => { + await register(); + const { db } = await import('../../src/lib/server/db/index.js'); + const { destroyServer, processDestructions } = await import('../../src/lib/server/provisioning/instances.js'); + const { resumeCleanup } = await import('../../src/lib/server/provisioning/history.js'); + const { komodoInspectHealthy, komodoDestroyDeployment, komodoDeleteDeployment } = await import('../../src/lib/server/integrations/komodo.js'); + vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'exited', reason: 'stopped' }); + db().exec("CREATE TRIGGER fail_cleanup BEFORE DELETE ON servers BEGIN SELECT RAISE(ABORT, 'cleanup interrupted'); END"); + const job = destroyServer(server); + if (!job.ok || !job.deploymentId) throw new Error('Could not queue removal'); + await processDestructions(); + const failed = await detail(job.deploymentId); + expect(failed.deployment.state).toBe('failed'); + expect(failed.progress?.checkpoints).toContainEqual(expect.objectContaining({ key: 'delete-deployment' })); + expect(failed.progress?.checkpoints.some(point => point.key === 'cleanup')).toBe(false); + const failure = failed.events.find(event => event.kind === 'outcome')!; + db().exec('DROP TRIGGER fail_cleanup'); + vi.setSystemTime(Date.now() + 1000); + expect(resumeCleanup(job.deploymentId)).toBe(true); + await processDestructions(); + const completed = await detail(job.deploymentId); + expect(completed.deployment).toMatchObject({ id: job.deploymentId, state: 'done', server_exists: 0 }); + expect(completed.events).toContainEqual(failure); + expect(completed.events.some(event => event.kind === 'resumed')).toBe(true); + expect(completed.progress?.checkpoints).toContainEqual(expect.objectContaining({ key: 'cleanup' })); + expect(komodoDestroyDeployment).toHaveBeenCalledTimes(1); + expect(komodoDeleteDeployment).toHaveBeenCalledTimes(1); + }); + + it('links the actual removal dependency while later eligible creation bypasses it', async () => { + const { db } = await import('../../src/lib/server/db/index.js'); + const { beginDeployment } = await import('../../src/lib/server/provisioning/history.js'); + const { queueCreateServer, processCreations } = await import('../../src/lib/server/provisioning/instances.js'); + const input = { gameMode: 'zombie' as const, mapSlug: 'theater', plutoKey: 'saved-key' }; + const first = queueCreateServer(input), second = queueCreateServer(input); + if (!first.ok || !second.ok) throw new Error('Could not queue creation'); + const removal = beginDeployment('old-instance', 'destroy'); + db().prepare('INSERT INTO server_creation_dependencies VALUES (?, ?)').run(first.deploymentId, removal); + await processCreations(); + expect((await detail(first.deploymentId)).progress?.blocker).toMatchObject({ code: 'removal-dependency', href: `/provisioning/${removal}` }); + expect((await detail(first.deploymentId)).deployment.state).toBe('queued'); + expect((await detail(second.deploymentId)).deployment.state).toBe('awaiting-health'); + }); + + it.each(['create', 'delete'] as const)('records uncertainty when the %s reply throws without inventing a successful checkpoint', async command => { + const komodo = await import('../../src/lib/server/integrations/komodo.js'); + const { queueCreateServer, processCreations, destroyServer, processDestructions } = await import('../../src/lib/server/provisioning/instances.js'); + let id: string; + if (command === 'create') { + vi.mocked(komodo.komodoCreateDeployment).mockRejectedValueOnce(new Error('reply lost')); + const job = queueCreateServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'saved-key' }); + if (!job.ok) throw new Error(job.message); + id = job.deploymentId; + await processCreations(); + expect(komodo.komodoCreateDeployment).toHaveBeenCalledTimes(1); + } else { + await register(); + vi.mocked(komodo.komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'exited', reason: 'stopped' }); + vi.mocked(komodo.komodoDeleteDeployment).mockRejectedValueOnce(new Error('reply lost')); + const job = destroyServer(server); + if (!job.ok || !job.deploymentId) throw new Error('Could not queue removal'); + id = job.deploymentId; + await processDestructions(); + expect(komodo.komodoDeleteDeployment).toHaveBeenCalledTimes(1); + } + const failed = await detail(id); + expect(failed.deployment.state).toBe('failed'); + expect(failed.progress?.command).toMatchObject({ state: 'uncertain', message: expect.stringContaining('reply lost') }); + expect(failed.progress?.checkpoints.some(point => point.key === (command === 'create' ? 'create-deployment' : 'delete-deployment'))).toBe(false); + }); + + it.each(['destroy', 'rebuild'] as const)('retains the resumable %s checkpoint when a beta arrives after confirmed remote deletion', async action => { + await register(); + const { db } = await import('../../src/lib/server/db/index.js'); + const { createScript } = await import('../../src/lib/server/fleet/scripts.js'); + const { destroyServer, processDestructions } = await import('../../src/lib/server/provisioning/instances.js'); + const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js'); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + const { resumeCleanup, resumeRebuild, REMOVING_SERVER } = await import('../../src/lib/server/provisioning/history.js'); + const { komodoInspectHealthy, komodoDestroyDeployment, komodoDeleteDeployment } = await import('../../src/lib/server/integrations/komodo.js'); + vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'exited', reason: 'stopped' }); + vi.mocked(komodoDeleteDeployment).mockImplementationOnce(async () => { + const { script, version } = createScript({ name: 'Late beta', targetRelPath: 'beta.gsc', scopeType: 'game', scopeValue: 't5', content: Buffer.from('beta') }); + const beta = db().prepare("INSERT INTO betas (script_id,script_version_id,trigger_kind,state) VALUES (?,?,'manual','running')").run(script.id, version.id).lastInsertRowid; + db().prepare("INSERT INTO beta_servers (beta_id,server,had_original,stage) VALUES (?,?,0,'ready')").run(beta, server); + return { ok: true }; + }); + const job = action === 'destroy' ? destroyServer(server) : requestLifecycle(server, 'rebuild'); + if (!job.ok || !job.deploymentId) throw new Error('Could not queue removal'); + await (action === 'destroy' ? processDestructions() : processProvisioning()); + const failed = await detail(job.deploymentId); + expect(failed.deployment).toMatchObject({ state: 'failed', stage: REMOVING_SERVER }); + expect(failed.progress?.checkpoints).toContainEqual(expect.objectContaining({ key: 'delete-deployment' })); + expect(action === 'destroy' ? failed.canResumeCleanup : failed.canResumeRebuild).toBe(true); + db().prepare("UPDATE betas SET state = 'rolled-back'").run(); + expect(action === 'destroy' ? resumeCleanup(job.deploymentId) : resumeRebuild(job.deploymentId)).toBe(true); + await (action === 'destroy' ? processDestructions() : processProvisioning()); + expect((await detail(job.deploymentId)).deployment.state).toBe(action === 'destroy' ? 'done' : 'queued'); + expect(komodoDestroyDeployment).toHaveBeenCalledTimes(1); + expect(komodoDeleteDeployment).toHaveBeenCalledTimes(1); + }); + + it('ignores occupancy evidence that returns after the request was cancelled', async () => { + await register(); + const { requestLifecycle, cancelProvisioning } = await import('../../src/lib/server/provisioning/requests.js'); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js'); + const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js'); + const job = requestLifecycle(server, 'restart'); + if (!job.ok) throw new Error(job.error); + vi.mocked(rconGetPlayerCount).mockImplementationOnce(async () => { + expect(cancelProvisioning(job.deploymentId)).toBe(true); + return 4; + }); + await processProvisioning(); + const cancelled = await detail(job.deploymentId); + expect(cancelled.deployment.state).toBe('cancelled'); + expect(cancelled.progress).toMatchObject({ checks: [], blocker: null }); + expect(komodoRestart).not.toHaveBeenCalled(); + }); + + it('does not attach replacement health to a deleted creation after a delayed probe', async () => { + const { db } = await import('../../src/lib/server/db/index.js'); + const { queueCreateServer, processCreations } = await import('../../src/lib/server/provisioning/instances.js'); + const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js'); + const { rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js'); + const job = queueCreateServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'saved-key' }); + if (!job.ok) throw new Error(job.message); + await processCreations(); + vi.mocked(komodoInspectHealthy).mockImplementationOnce(async () => { + db().prepare('DELETE FROM servers WHERE name = ?').run(server); + await register(); + return { ok: true, state: 'running' }; + }); + await processCreations(); + const old = await detail(job.deploymentId); + expect(old.deployment.server_deleted_at).toBeTruthy(); + expect(old.progress?.checks).toEqual([]); + expect(old.progress?.checkpoints.some(point => point.key.startsWith('verify-'))).toBe(false); + expect(rconStatusResult).not.toHaveBeenCalled(); + }); +}); diff --git a/tests/unit/provisioning-presentation.test.ts b/tests/unit/provisioning-presentation.test.ts new file mode 100644 index 0000000..df6b962 --- /dev/null +++ b/tests/unit/provisioning-presentation.test.ts @@ -0,0 +1,131 @@ +import { describe, expect, it } from 'vitest'; +import type { ProvisioningDetail, ProvisioningProgress } from '../../src/lib/provisioning-detail.js'; +import { provisioningActions } from '../../src/lib/provisioning.js'; +import { isProvisioningDetail, isProvisioningEventPage, provisioningGroups, provisioningOutcome, provisioningReturn } from '../../src/lib/ui/provisioning-presentation.js'; + +const at = '2026-09-28T12:00:00Z'; +function progress(patch: Partial = {}): ProvisioningProgress { + return { version: 1, phase: 'verify-map', phaseStartedAt: at, observedAt: at, blocker: null, checks: [], checkpoints: [], + command: null, recovery: null, recipeStatus: null, ...patch }; +} +function fixture(): ProvisioningDetail { + return { + deployment: { id: 'operation-a', server: 't5_zombie_theater_1', action: 'create', state: 'awaiting-health', stage: 'Waiting for a map', + server_deleted_at: null, server_exists: 1, started_at: at, finished_at: null, triggered_by: 'ui', error: null, warnings: '[]' }, + generatedAt: at, events: [], activity: { total: 0, latestId: null, failureExcerpt: null }, sources: [], + recipe: null, blueprintName: null, canResumeCleanup: false, canResumeRebuild: false, canResumeVerification: false, canCancelRemoval: false, + collection: null, progress: progress(), capacity: { evaluatedAt: at, pools: [] }, warningMessages: [] + }; +} + +describe('provisioning outcome and evidence presentation', () => { + it.each(provisioningActions)('describes cancelled %s as terminal, with no automatic execution claim', action => { + const detail = fixture(); + detail.deployment.action = action; + detail.deployment.state = 'cancelled'; + const outcome = provisioningOutcome(detail); + expect(outcome.active).toBe(false); + expect(outcome.headline).toContain('cancelled'); + expect(outcome.description).toContain('remaining work will not run'); + expect(outcome.description).not.toMatch(/verifying|automatically/); + }); + + it('keeps backend-authorized cleanup actionable and active after identity deletion', () => { + const detail = fixture(); + detail.deployment.action = 'destroy'; + detail.deployment.state = 'failed'; + detail.deployment.server_deleted_at = at; + detail.deployment.server_exists = 0; + detail.canResumeCleanup = true; + expect(provisioningOutcome(detail).description).toContain('Resume the remaining local cleanup'); + detail.deployment.state = 'waiting'; + detail.canResumeCleanup = false; + detail.progress = progress({ phase: 'cleanup', checks: [] }); + expect(provisioningOutcome(detail).active).toBe(true); + expect(provisioningOutcome(detail).description).toContain('without repeating container removal'); + expect(provisioningOutcome(detail).description).not.toContain('eligibility'); + }); + + it('does not invent completed phases from a later phase or final outcome', () => { + const detail = fixture(); + const steps = provisioningGroups(detail).flatMap(group => group.steps); + expect(steps.find(step => step.key === 'prepare')?.state).toBe('unknown'); + expect(steps.find(step => step.key === 'verify-map')?.state).toBe('current'); + detail.deployment.state = 'done'; + expect(provisioningGroups(detail).flatMap(group => group.steps).every(step => step.state === 'unknown')).toBe(true); + detail.progress = null; + expect(provisioningGroups(detail)).toEqual([]); + expect(provisioningOutcome(detail).description).toContain('evidence was not recorded'); + }); + + it('shows passed container and telemetry game checks while waiting on the engine map', () => { + const detail = fixture(); + detail.progress = progress({ checks: [ + { key: 'container', label: 'Container readiness', result: 'pass', source: 'komodo', checkedAt: at, receivedAt: null, expected: 'healthy', actual: 'healthy', message: null }, + { key: 'game', label: 'Game status', result: 'pass', source: 'telemetry', checkedAt: at, receivedAt: at, expected: null, actual: null, message: 'Fresh telemetry accepted' }, + { key: 'map', label: 'Engine map', result: 'waiting', source: 'telemetry', checkedAt: at, receivedAt: at, expected: 'zombie_theater', actual: 'zombie_moon', message: null } + ] }); + const steps = provisioningGroups(detail).flatMap(group => group.steps); + expect(steps.find(step => step.key === 'verify-container')?.state).toBe('done'); + expect(steps.find(step => step.key === 'verify-game')?.state).toBe('done'); + expect(steps.find(step => step.key === 'verify-map')?.state).toBe('waiting'); + expect(steps.find(step => step.key === 'verify-game')?.label).not.toContain('RCON'); + detail.deployment.state = 'done'; + expect(provisioningOutcome(detail).description).not.toContain('RCON'); + }); + + it('newer unknown readiness evidence supersedes an old completion checkpoint', () => { + const detail = fixture(); + detail.progress = progress({ checkpoints: [{ key: 'verify-game', label: 'Game passed', at, phase: 'verify-game' }], + checks: [{ key: 'game', label: 'Game status', result: 'unknown', source: null, checkedAt: '2026-09-28T12:01:00Z', receivedAt: null, expected: null, actual: null, message: 'No fresh evidence' }] }); + expect(provisioningGroups(detail).flatMap(group => group.steps).find(step => step.key === 'verify-game')).toMatchObject({ state: 'unknown', at: '2026-09-28T12:01:00Z' }); + }); + + it('preserves the failed request when later recovery checks pass', () => { + const detail = fixture(); + detail.deployment.action = 'restart'; detail.deployment.state = 'failed'; + detail.progress = progress({ phase: 'verify-game', recovery: { state: 'released', at, message: 'Recovery verified and capacity released.' }, + checks: [{ key: 'game', label: 'Game status', result: 'pass', source: 'rcon', checkedAt: at, receivedAt: null, expected: null, actual: 0, message: null }] }); + expect(provisioningOutcome(detail).headline).toBe('Restart failed'); + expect(provisioningGroups(detail).flatMap(group => group.steps).find(step => step.key === 'verify-game')?.state).toBe('done'); + }); + + it('keeps removal checkpoints separate from replacement creation', () => { + const detail = fixture(); detail.deployment.action = 'rebuild'; detail.deployment.state = 'waiting'; + detail.progress = progress({ phase: 'cleanup', checkpoints: [{ key: 'delete-deployment', label: 'Remote deployment deleted', at, phase: 'delete-deployment' }] }); + const groups = provisioningGroups(detail); + expect(groups.map(group => group.key)).toEqual(['request', 'remove', 'create', 'verify']); + expect(groups.flatMap(group => group.steps).find(step => step.key === 'delete-deployment')?.state).toBe('done'); + expect(groups.flatMap(group => group.steps).find(step => step.key === 'cleanup')?.state).toBe('waiting'); + expect(groups.flatMap(group => group.steps).find(step => step.key === 'prepare')?.state).toBe('pending'); + }); +}); + +describe('provisioning response and return-context boundaries', () => { + it('retains list filters without accepting an external or unrelated return destination', () => { + expect(provisioningReturn('/provisioning?view=history&action=restart&server=Lobby+1&limit=50&redirect=https://evil.test').href) + .toBe('/provisioning?view=history&action=restart&server=Lobby+1&limit=50'); + for (const value of ['https://evil.test', '//evil.test', '/\\evil.test', '/\n/evil.test', '/settings', '/provisioning/other']) { + expect(provisioningReturn(value).href).toBe('/provisioning'); + } + expect(provisioningReturn('/collections/launches/launch-a').href).toBe('/collections/launches/launch-a'); + }); + + it('rejects incomplete nested snapshots before they replace a working detail', () => { + const detail = fixture(); + expect(isProvisioningDetail(detail, detail.deployment.id)).toBe(true); + expect(isProvisioningDetail(detail, 'another-id')).toBe(false); + for (const patch of [ + { progress: undefined }, { progress: {} }, { progress: { ...detail.progress, checks: [null] } }, + { capacity: { evaluatedAt: at, pools: [{ kind: 'restart', holders: null }] } }, { activity: { total: 0 } }, + { warningMessages: {} }, { recipe: { files: [] } }, { sources: [{ kind: 'campaign', label: 'Campaign', source_id: 'a', href: 'javascript:alert(1)' }] } + ]) expect(isProvisioningDetail({ ...detail, ...patch }, detail.deployment.id)).toBe(false); + expect(isProvisioningDetail({ ...detail, progress: { ...detail.progress, command: { state: 'rejected', at, message: 'Rejected before submission' } } }, detail.deployment.id)).toBe(true); + }); + + it('rejects an invalid history event without dropping the existing history view', () => { + const page = { events: [{ id: 1, created_at: at, message: 'Queued' }], total: 1, totalEvents: 1, latestId: 1, watermark: 1, nextCursor: null }; + expect(isProvisioningEventPage(page)).toBe(true); + expect(isProvisioningEventPage({ ...page, events: [{ id: 1, created_at: 'invalid', message: 'Queued' }] })).toBe(false); + }); +}); diff --git a/tests/unit/provisioning-progress.test.ts b/tests/unit/provisioning-progress.test.ts new file mode 100644 index 0000000..804aa94 --- /dev/null +++ b/tests/unit/provisioning-progress.test.ts @@ -0,0 +1,111 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import Database from 'better-sqlite3'; +import { readFile } from 'node:fs/promises'; +import { newFixture, type Fixture } from '../helpers/fixture.js'; + +let fixture: Fixture; +beforeEach(async () => { + vi.resetModules(); + fixture = await newFixture([]); + vi.useFakeTimers({ toFake: ['Date'] }); + vi.setSystemTime(new Date('2026-09-28T12:00:00Z')); +}); +afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); }); + +async function setup() { + const { db } = await import('../../src/lib/server/db/index.js'); + const handle = db(); + handle.prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by) + VALUES ('operation', 'server', 'create', 'waiting', 'Recorded stage', '2026-09-28T12:00:00Z', 'test')`).run(); + const { recordProgress, readProgress } = await import('../../src/lib/server/provisioning/progress.js'); + const read = () => readProgress(handle.prepare('SELECT observation_json FROM server_deployments WHERE id = ?').get('operation') as { observation_json: string }); + return { handle, recordProgress, read }; +} + +describe('provisioning evidence persistence', () => { + it('keeps phase duration stable and avoids semantic event spam during repeated occupancy checks', async () => { + const { handle, recordProgress, read } = await setup(); + for (let players = 4; players > 0; players--) { + recordProgress('operation', { phase: 'eligibility', blocker: { code: 'occupied', message: `${players} players remain`, automatic: true }, + checks: [{ key: 'occupancy', label: 'Players', source: 'telemetry', result: 'waiting', actual: players, + receivedAt: new Date().toISOString() }] }); + vi.setSystemTime(Date.now() + 15_000); + } + expect(read()).toMatchObject({ phaseStartedAt: '2026-09-28T12:00:00.000Z', observedAt: '2026-09-28T12:00:45.000Z', + checks: [expect.objectContaining({ actual: 1, receivedAt: '2026-09-28T12:00:45.000Z' })] }); + expect(handle.prepare("SELECT count(*) AS count FROM server_deployment_events WHERE kind = 'blocker'").get()).toEqual({ count: 1 }); + recordProgress('operation', { phase: 'container-action', blocker: null, + checkpoint: { key: 'eligibility', label: 'Eligibility checked', phase: 'eligibility' } }); + expect(read()).toMatchObject({ phaseStartedAt: '2026-09-28T12:01:00.000Z', blocker: null }); + }); + + it('replaces transient checks while retaining confirmed milestones and prior command uncertainty', async () => { + const { handle, recordProgress, read } = await setup(); + recordProgress('operation', { phase: 'verify-game', command: { state: 'uncertain', message: 'Reply was lost' }, + checkpoint: { key: 'container-action', label: 'Submission attempted', phase: 'container-action' }, + checks: [{ key: 'container', label: 'Container', source: 'komodo', result: 'pass' }, + { key: 'game', label: 'Game', source: 'telemetry', result: 'pass' }] }); + vi.setSystemTime(Date.now() + 15_000); + recordProgress('operation', { phase: 'verify-container', checks: [{ key: 'container', label: 'Container', source: 'komodo', result: 'unknown' }] }); + expect(read()?.checks).toHaveLength(1); + expect(read()?.checks[0].result).toBe('unknown'); + expect(read()?.checkpoints).toHaveLength(1); + expect(read()?.command?.state).toBe('uncertain'); + expect(handle.prepare("SELECT message FROM server_deployment_events WHERE kind = 'command'").get()).toEqual({ message: 'Reply was lost' }); + }); + + it('keeps operation-owned recovery evidence after slot/server removal and does not rewrite the failed outcome', async () => { + const { handle, recordProgress, read } = await setup(); + handle.prepare("UPDATE server_deployments SET state = 'failed', finished_at = '2026-09-28T12:00:00Z' WHERE id = 'operation'").run(); + recordProgress('operation', { recovery: { state: 'held', message: 'Observing possible submission' } }); + vi.setSystemTime(Date.now() + 30_000); + recordProgress('operation', { recovery: { state: 'released', message: 'Recovery verified' } }); + recordProgress('operation', { recovery: { state: 'released', message: 'Recovery verified' } }); + expect(read()?.recovery).toMatchObject({ state: 'released', at: '2026-09-28T12:00:30.000Z' }); + expect(handle.prepare("SELECT state, finished_at FROM server_deployments WHERE id = 'operation'").get()) + .toEqual({ state: 'failed', finished_at: '2026-09-28T12:00:00Z' }); + expect(handle.prepare("SELECT count(*) AS count FROM server_deployment_events WHERE kind = 'recovery'").get()).toEqual({ count: 2 }); + }); + + it('rolls back state, snapshot, and milestone events together with its caller transaction', async () => { + const { handle, recordProgress, read } = await setup(); + expect(() => handle.transaction(() => { + handle.prepare("UPDATE server_deployments SET state = 'running' WHERE id = 'operation'").run(); + recordProgress('operation', { phase: 'cleanup', checkpoint: { key: 'delete-deployment', label: 'Remote deletion confirmed' } }); + throw new Error('Interrupted transaction'); + })()).toThrow('Interrupted transaction'); + expect(read()).toBeNull(); + expect(handle.prepare("SELECT state FROM server_deployments WHERE id = 'operation'").get()).toEqual({ state: 'waiting' }); + expect(handle.prepare('SELECT * FROM server_deployment_events').all()).toEqual([]); + }); + + it('treats unsupported or malformed historical snapshots as unrecorded', async () => { + const { readProgress } = await import('../../src/lib/server/provisioning/progress.js'); + for (const observation_json of [null, '', '[]', '{broken', '{"version":2}', '{"version":1,"checks":"pass"}']) { + expect(readProgress({ observation_json })).toBeNull(); + } + }); + + it('adds nullable evidence columns without losing historical identity, row order, or events across reopen', async () => { + const schema = (await readFile(new URL('../../src/lib/server/db/schema.sql', import.meta.url), 'utf8')) + .replace(',\n phase_key TEXT,\n phase_started_at TEXT,\n observed_at TEXT,\n observation_json TEXT', '') + .replace(',\n kind TEXT,\n phase_key TEXT,\n details_json TEXT', '') + .replace("'create', 'start', 'restart', 'rebuild', 'stop', 'destroy'", "'create', 'destroy'"); + const old = new Database(fixture.dbPath); + old.exec(schema); + old.prepare(`INSERT INTO server_deployments (rowid,id,server,server_deleted_at,action,state,stage,started_at,triggered_by) + VALUES (42,'legacy','server','2026-09-01','create','done','Recorded result','2026-08-01','test')`).run(); + old.prepare("INSERT INTO server_deployment_events (id,deployment_id,created_at,message) VALUES (19,'legacy','2026-08-01','Original evidence')").run(); + old.close(); + const { db, closeDb } = await import('../../src/lib/server/db/index.js'); + expect(db().prepare("SELECT rowid, server_deleted_at, observation_json FROM server_deployments WHERE id = 'legacy'").get()) + .toEqual({ rowid: 42, server_deleted_at: '2026-09-01', observation_json: null }); + const { recordProgress, readProgress } = await import('../../src/lib/server/provisioning/progress.js'); + recordProgress('legacy', { recovery: { state: 'unknown', message: 'Historical slot release was not recorded' } }); + closeDb(); + const saved = db().prepare("SELECT observation_json FROM server_deployments WHERE id = 'legacy'").get() as { observation_json: string }; + expect(readProgress(saved)?.recovery?.state).toBe('unknown'); + expect(db().prepare('SELECT id, message FROM server_deployment_events WHERE id = 19').get()).toEqual({ id: 19, message: 'Original evidence' }); + expect(db().pragma('foreign_key_check')).toEqual([]); + }); +}); diff --git a/tests/unit/provisioning-refresh.test.ts b/tests/unit/provisioning-refresh.test.ts new file mode 100644 index 0000000..bbb3028 --- /dev/null +++ b/tests/unit/provisioning-refresh.test.ts @@ -0,0 +1,119 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { createProvisioningRefresh, type ProvisioningRefreshState } from '../../src/lib/ui/provisioning-refresh.js'; + +type Detail = { id: string; revision: number }; +const detail = (id: string, revision = 1): Detail => ({ id, revision }); +const response = (id: string, revision = 1) => Response.json(detail(id, revision)); +const controllers: { dispose: () => void }[] = []; +afterEach(() => { controllers.splice(0).forEach(controller => controller.dispose()); vi.useRealTimers(); }); +function setup(fetcher: typeof fetch) { + let state!: ProvisioningRefreshState; + const onChange = vi.fn(value => { state = value; }); + const controller = createProvisioningRefresh({ fetcher, onChange, timeoutMs: 100, + validate: (value, id): value is Detail => !!value && typeof value === 'object' && (value as Detail).id === id, + now: () => '2026-09-28T12:00:00Z' }); + controllers.push(controller); + return { ...controller, onChange, state: () => state }; +} +function deferred() { + let resolve!: (value: Response) => void; + let reject!: (error: Error) => void; + const promise = new Promise((done, fail) => { resolve = done; reject = fail; }); + return { promise, resolve, reject }; +} + +describe('provisioning detail refresh ownership', () => { + it('uses the server snapshot without another initial request and coalesces background reads', async () => { + const pending = deferred(); + const fetcher = vi.fn(() => pending.promise); + const page = setup(fetcher); + page.setOperation('a', detail('a'), '2026-09-28T11:59:00Z'); + expect(fetcher).not.toHaveBeenCalled(); + const first = page.refresh(); + const second = page.refresh(); + expect(fetcher).toHaveBeenCalledTimes(1); + pending.resolve(response('a', 2)); + await Promise.all([first, second]); + expect(page.state()).toMatchObject({ detail: detail('a', 2), refreshing: false, refreshedAt: '2026-09-28T12:00:00Z' }); + }); + + it('ignores obsolete success and failure across A → B → A, even when transport ignores abort', async () => { + const oldA = deferred(), oldB = deferred(), newA = deferred(); + const fetcher = vi.fn().mockReturnValueOnce(oldA.promise).mockReturnValueOnce(oldB.promise).mockReturnValueOnce(newA.promise); + const page = setup(fetcher); + page.setOperation('a', detail('a')); + const first = page.refresh(); + page.setOperation('b', detail('b')); + const second = page.refresh(); + page.setOperation('a', detail('a', 3)); + const third = page.refresh(); + newA.resolve(response('a', 4)); + await third; + oldA.resolve(response('a', 0)); + oldB.reject(new Error('Obsolete failure')); + await Promise.all([first, second]); + expect(page.state()).toMatchObject({ id: 'a', detail: detail('a', 4), loadError: '', refreshing: false }); + }); + + it('invalidates a pending poll and reconciles an accepted mutation before old data can return', async () => { + const old = deferred(); + const fetcher = vi.fn().mockReturnValueOnce(old.promise).mockResolvedValueOnce(Response.json({ ok: true })) + .mockResolvedValueOnce(response('a', 3)); + const page = setup(fetcher); + page.setOperation('a', detail('a')); + const polling = page.refresh(); + await page.act('resume', 'Verification resumed.'); + old.resolve(response('a', 2)); + await polling; + expect(fetcher).toHaveBeenCalledTimes(3); + expect(page.state()).toMatchObject({ detail: detail('a', 3), notice: 'Verification resumed.', busy: null, loadError: '', actionError: '' }); + }); + + it('retains accepted-action feedback if reconciliation fails, and action errors survive successful polls', async () => { + const fetcher = vi.fn().mockResolvedValueOnce(Response.json({ ok: true })) + .mockRejectedValueOnce(new Error('Offline')) + .mockResolvedValueOnce(Response.json({ error: 'Removal already began.' }, { status: 409 })) + .mockResolvedValueOnce(response('a', 2)).mockResolvedValueOnce(response('a', 3)); + const page = setup(fetcher); + page.setOperation('a', detail('a')); + await page.act('resume', 'Resume accepted.'); + expect(page.state()).toMatchObject({ notice: 'Resume accepted.', loadError: 'Offline', actionError: '', detail: detail('a') }); + await page.act('cancel', 'Cancelled.'); + await page.refresh(); + expect(page.state()).toMatchObject({ actionError: 'Removal already began.', notice: '', loadError: '', detail: detail('a', 3) }); + }); + + it('does not duplicate mutations or poll during them and ignores completion after navigation', async () => { + const action = deferred(); + const fetcher = vi.fn(() => action.promise); + const page = setup(fetcher); + page.setOperation('a', detail('a')); + const first = page.act('resume', 'Resumed.'); + await page.act('resume', 'Resumed twice.'); + await page.refresh(); + expect(fetcher).toHaveBeenCalledTimes(1); + page.setOperation('b', detail('b')); + action.resolve(Response.json({ ok: true })); + await first; + expect(fetcher).toHaveBeenCalledTimes(1); + expect(page.state()).toMatchObject({ id: 'b', busy: null, notice: '', actionError: '' }); + }); + + it('times out reads without discarding the last snapshot and suppresses updates after unmount', async () => { + vi.useFakeTimers(); + const fetcher = vi.fn((_url, init) => new Promise((_, reject) => { + init?.signal?.addEventListener('abort', () => reject(new Error('aborted'))); + })) as typeof fetch; + const page = setup(fetcher); + page.setOperation('a', detail('a')); + const pending = page.refresh(); + await vi.advanceTimersByTimeAsync(100); + await pending; + expect(page.state()).toMatchObject({ detail: detail('a'), loadError: 'Operation refresh timed out. Try again.', refreshing: false }); + const next = page.refresh(); + page.dispose(); + const calls = page.onChange.mock.calls.length; + await next; + expect(page.onChange).toHaveBeenCalledTimes(calls); + }); +}); diff --git a/tests/unit/provisioning.test.ts b/tests/unit/provisioning.test.ts index 6d9e71f..9422fce 100644 --- a/tests/unit/provisioning.test.ts +++ b/tests/unit/provisioning.test.ts @@ -303,7 +303,8 @@ describe('Watchdog provisioning and telemetry evidence', () => { expect(result.ok).toBe(true); if (!result.ok) throw new Error(result.error); expect(requestWatchdogLifecycle(server, 'restart', 'incident')).toEqual(result); - expect(getDeployment(result.deploymentId)?.sources).toEqual([{ kind: 'watchdog', source_id: 'incident', server }]); + expect(getDeployment(result.deploymentId)?.sources).toEqual([{ kind: 'watchdog', source_id: 'incident', server, + label: 'Server down', href: '/maintenance/incident' }]); expect(await operation(result.deploymentId)).toMatchObject({ triggered_by: 'watchdog:incident' }); db().prepare("INSERT INTO settings (key,value) VALUES ('watchdog_enabled','0')").run(); await processProvisioning(); diff --git a/tests/unit/restart-rollout.test.ts b/tests/unit/restart-rollout.test.ts index da04ded..8c59e23 100644 --- a/tests/unit/restart-rollout.test.ts +++ b/tests/unit/restart-rollout.test.ts @@ -348,7 +348,8 @@ describe('Provisioning ownership', () => { const source = db().prepare('SELECT provisioning_id FROM deployment_servers WHERE deployment_id = ?').get('campaign') as { provisioning_id: string }; const id = source.provisioning_id; expect(provisioningOperation(id)).toMatchObject({ action: 'restart', state: 'waiting', submitted_at: null }); - expect(getDeployment(id)?.sources).toEqual([{ kind: 'campaign', source_id: 'campaign', server: servers[0] }]); + expect(getDeployment(id)?.sources).toEqual([{ kind: 'campaign', source_id: 'campaign', server: servers[0], + label: 'Promotion campaign', href: '/campaigns/campaign' }]); expect(komodoRestart).not.toHaveBeenCalled(); expect(komodoInspectHealthy).not.toHaveBeenCalled(); expect(rconGetPlayerCount).not.toHaveBeenCalled();