Complete provisioning job detail pages and lifecycle evidence

2026-09-28 14:17:19 -04:00
parent 5b65520bbc
commit 3fb59cd598
41 changed files with 2866 additions and 346 deletions
@@ -0,0 +1,44 @@
+# Provisioning detail implementation
+
+Implemented September 28, 2026, following the [approved review plan](provisioning-improvement-plan.md).
+
+## Delivered behavior
+
+The job page now leads with identity, recorded outcome, blocker, and permitted recovery action. Grouped execution phases and timestamped evidence precede supporting metadata and launch files. Mobile follows the same reading order. Operation IDs and links are copyable, and the return link preserves Provisioning filters or leads back to related work.
+
+The worker records versioned progress snapshots and semantic events at existing decision points. Container, game, map, and occupancy checks identify their source and observation time. Durable milestones preserve remote deletion, local cleanup, command certainty, and operation-owned recovery capacity. A failed restart can retain its failed outcome while later recovery evidence records capacity release. Legacy records retain their original stage and outcome without invented checkpoints.
+
+Cleanup resumes honor backend capabilities even for deleted identities. Confirmed remote deletion and the existing cleanup-resume stage are saved together before local cleanup, preserving recovery if a later guard or local operation fails. Late observations are rejected after cancellation, deletion, or replacement. Existing occupancy rules, scheduling, command counts, and observation-only resumes are covered by regression tests.
+
+The page uses server-loaded detail and abortable, visit-scoped refreshes. Action feedback survives background refresh failures; outdated requests cannot replace another job's detail. Terminal records continue visibility-aware polling because recovery, deletion, and capabilities can still change.
+
+Five recent events are visible. Full history loads in 25-entry pages with literal search, ordering, and a fixed upper event ID. Recent activity stays still while full history is open; new events appear through an explicit Jump to latest control. The legacy detail API still returns full events by default. The new page opts into `?events=recent`.
+
+Queued recipe captures and files selected for an attempt have separate labels and timestamps. Neither implies successful installation. Related work names resolve when available, while missing source IDs remain visible without broken links.
+
+## Validation
+
+- Full suite: **898 tests passed across 72 files**.
+- Svelte/TypeScript: `npm run check` passed with **0 errors and 0 warnings**.
+- Production: `npm run build` passed with the Node adapter.
+- Database tests cover additive migration, historical identity and event preservation, transaction rollback, stable phase timing, semantic outcomes across same-ID resumes, and full versus bounded API compatibility.
+- Lifecycle tests cover ambiguous and thrown remote replies, telemetry and map evidence, cancelled/replaced identity races, exact recovery ownership, and destroy/rebuild cleanup after confirmed deletion without repeating remote commands.
+- Refresh/presentation tests cover navigation and action races, persistent acknowledged action results, failed reconciliation, cancellation, deleted cleanup, legacy evidence, and malformed responses.
+- Firefox checks exercised history pagination, literal search, stable new-event handling, failed-refresh retention and recovery, and layouts at 1440, 390, and 320 pixels without page-wide overflow. Enter and Space operate phase disclosures. A deleted-identity cleanup resume completed through the real local API; retained restart recovery, cancellation, and legacy evidence also rendered correctly.
+- Reflow also passed at 720 CSS pixels with pixel ratio 2, and long unbroken identifiers fit at 320px. This simulates the available layout width of 200% zoom on a 1440px display; native browser zoom was not measured. A separate CSS-zoom probe kept the detail content in bounds but exposed overflow in the existing global navigation.
+
+Browser fixtures use a temporary database and disabled external integrations. They exercise the real page and local detail/history APIs; lifecycle tests stub remote systems. This is not a production fleet smoke test or a formal accessibility conformance audit. The build reports the repository's existing `csrf.checkOrigin` deprecation notice.
+
+## Measurements and screenshots
+
+For a synthetic 5,001-event operation, full detail serialized to **1,884,119 bytes** and bounded detail to **3,011 bytes**. Thirty local bounded reads averaged **2.93 ms**. These measurements establish payload reduction on the local fixture, not a production latency guarantee; event totals and search still query retained history.
+
+In the 1440px create/map-wait fixture, the execution heading moved from approximately **1,073px** in the original composition to **675px**. The page keeps the outcome above execution and the recipe in a supporting column. Phone layouts move context below execution and activity.
+
+Updated screenshots: [desktop create/map wait](provisioning-review-assets/implemented-create-desktop.png), [mobile create/map wait](provisioning-review-assets/implemented-create-mobile.png), [deleted identity with permitted cleanup](provisioning-review-assets/implemented-deleted-cleanup.png), and [failed restart with retained recovery](provisioning-review-assets/implemented-restart-recovery.png). The original review captures remain linked from the approved plan.
+
+## Compatibility and rollout
+
+The schema additions are nullable, and old stage text and default API event responses remain supported. Progress is written as new work is evaluated; historical operations without structured evidence use the legacy presentation. Detail reads perform no remote probes.
+
+The change is implemented in the workspace. Normal deployment should verify one newly instrumented job and one legacy record against the configured integrations. No production deployment was performed during this work.
@@ -0,0 +1,230 @@
+# Provisioning job detail improvement plan
+
+Review date: September 28, 2026. Source baseline: `5b65520`.
+
+The recommendation is to make `/provisioning/:id` a complete operational record: identify the job, explain its current outcome and blocker, expose the evidence behind that outcome, and show the next permitted action. Progress and recovery should precede historical launch files.
+
+Approved and implemented on September 28, 2026. The original review and agreed scope remain below; see the [implementation and validation record](provisioning-implementation.md) for delivered behavior, checks, and updated screenshots.
+
+## Review method and participants
+
+The requested panel covered three UI/UX roles, two CI/CD systems engineering roles, and one software engineering role. The platform allowed only three child-agent threads and rejected additional spawns even after those reviews completed. The three threads therefore performed two separate rounds, first UX and then engineering. This provides six role-based reviews, rather than six independent agents. Each round included direct peer discussion; engineering reviewed and challenged the UX proposals.
+
+| Review role | Proposal and contribution |
+| --- | --- |
+| UI/UX: information architecture | Prioritize job identity, outcome, blocker, safe action, progress, activity, and broader work context. Separate historical completion from current server health. |
+| UI/UX: interaction and accessibility | Use state-specific recovery actions, stable refresh behavior, persistent action feedback, keyboard-accessible progress, and usable history while updates arrive. |
+| UI/UX: visual design | Replace the overloaded summary and wrapping step chips with a compact outcome card, grouped vertical progress, and supporting metadata. Reuse the existing design system. |
+| CI/CD systems: lifecycle and queues | Preserve destructive checkpoints, occupancy gates, scheduling, command certainty, identity guards, and recovery-slot semantics. Make queue claims match actual admission behavior. |
+| CI/CD systems: observability and reliability | Record the source and age of verification evidence, distinguish page freshness from worker checks, preserve recovery evidence, and validate compatibility and failure scenarios. |
+| Software engineering | Define shared types and component boundaries, reuse server-loaded detail, fix route/poll/action races, and stage additive implementation with regression coverage. |
+
+The review covered the page, shared history component, lifecycle APIs, worker, queues, persistence, related collection/maintenance/campaign pages, and relevant tests. Root also inspected the current source in Firefox using an isolated temporary database, disabled integration credentials, and synthetic detail responses. Desktop captures used 1440px width; mobile used 390px. These are composed UI scenarios, not executions of real provisioning workflows or measurements of production use.
+
+Review assets: [create waiting for a map](provisioning-review-assets/create-map-wait-desktop.png), [mobile rebuild composition](provisioning-review-assets/rebuild-mobile.png), and [deleted identity with resumable cleanup](provisioning-review-assets/deleted-cleanup-desktop.png).
+
+## What feels incomplete today
+
+The page already has useful recovery buttons, server diagnostics links, a recorded launch recipe, source backlinks, elapsed time, progress steps, and searchable activity. Preserve those capabilities. The gaps concern their priority and the reliability of the story they tell.
+
+| Finding | Effect on the operator | Evidence |
+| --- | --- | --- |
+| One summary mixes status, actions, errors, warnings, diagnostics, map, timing, and provenance. The recipe appears before progress. | Monitoring and recovery compete with secondary information. | [Detail page](../src/routes/provisioning/[id]/+page.svelte), summary and recipe markup. In the synthetic desktop create case, the flow began about 1,073px down. |
+| Rebuild renders 12 equally weighted wrapping step chips. | Removal, cleanup, and fresh creation do not read as distinct phases. | [Detail page](../src/routes/provisioning/[id]/+page.svelte), `steps`; [OperationHistory](../src/lib/ui/components/OperationHistory.svelte). |
+| Progress and step duration depend on matching human-readable stage/event text. | New wording can move the highlighted step backward; changes in player count can appear to restart a step timer. | Detail page `currentStep` and `stageStarted`; [creation verification](../src/lib/server/provisioning/instances.ts) now emits “RCON or fresh telemetry” stages. |
+| Runtime cancellation is handled after runtime-specific explanation logic. | A cancelled start/restart/stop can still claim automatic verification is underway. | Detail page `explanation`. |
+| The deleted-server flag suppresses every cleanup button and the active-state presentation. | A permitted local cleanup can become inaccessible or appear inactive after resuming. | Detail page recovery controls and `active` versus [history.ts](../src/lib/server/provisioning/history.ts), `canResumeCleanup`. Missing button reproduced with a synthetic permitted response. |
+| Completion copy asserts RCON health in the present tense. | Telemetry-based verification is misreported, and historical success can look like current health. | Detail page explanations; [status-evidence.ts](../src/lib/server/fleet/status-evidence.ts), `probeServerStatus`. |
+| One refresh guard and one error string handle reads, mutations, and navigation. | A post-action reload can be dropped, an old request can affect another job, and polling can erase an action error. | Detail page `load`, `resume`, `cancelRemoval`; source-inferred race cases. |
+| A failed restart can still hold a recovery slot. | A simple failed badge does not explain continued observation or effects on other jobs. | [restarts.ts](../src/lib/server/provisioning/restarts.ts), `refreshRestartRollout`. |
+| Activity is collapsed, operation ID is absent from the page body, and related-work links are scattered. | Investigation and handoff require avoidable searching and navigation. | Detail page and shared history component. |
+
+## Proposed page
+
+Use one reading order across all six actions. At wide widths, metadata and launch inputs can occupy a supporting column. On phones they follow progress and activity in the DOM. Avoid a sticky sidebar.
+
+```text
+Provisioning / Restart server
+Origins · Lobby 1     t6_zombie_origins_1
+Operation ID [Copy]   [Copy link]                  [View server]
+
+[Waiting] Waiting for players to leave
+Last evaluated: 3 players · source and observation time
+Continues automatically after an eligible empty/stopped check.
+Relevant permitted action or dependency link
+Total elapsed · Time in phase · Page updated / Refresh
+
+EXECUTION AND EVIDENCE                     OPERATION CONTEXT
+  Eligibility                             Requested by / requested at
+  Container action                        Submission attempted / finished
+  Readiness verification                  Related campaign/beta/maintenance
+    Container                             Collection launch
+    Game status
+    Selected map, when required            Recorded launch files
+                                          [Launch another server]
+RECENT ACTIVITY
+  Five latest entries + relevant failure
+  [Full searchable history]
+```
+
+The example evidence requires the new contract below; do not populate it from guessed values.
+
+### Identity, outcome, and next action
+
+- Show the action, friendly map/lobby title, technical server name, and copyable operation ID/link. Use existing identity and copy components.
+- Lead with the specific blocker or outcome, followed by what happens next. Place the recorded error and permitted recovery control together. Give warnings a labeled explanation rather than color alone.
+- Treat deletion as historical identity context. It must not replace the outcome, hide an actionable cleanup, or link an old operation to a replacement server with the same name.
+- Determine active progress from the operation and recovery state, independently of whether a live server link is available.
+- Distinguish a recorded outcome from ongoing recovery. A failed request can show “Recovery is still being checked; restart capacity remains reserved” when the backend records that condition.
+- Use past-tense completion language with its timestamp: “Creation completed. Container readiness, game status, and the selected engine map were verified at …”. Name RCON or telemetry only when recorded. Current health remains on the server page.
+- Preserve an explicit return link to the originating Provisioning filters or parent work, with a safe local fallback. Group all requesting campaigns, betas, maintenance records, and collection launches under Related work. Keep retained source IDs visible when their originating record no longer exists, without a broken link.
+- Finishing a restart does not mean its campaign, beta, or maintenance verification has finished. Link back to that work with accurate wording.
+
+### Progress and evidence
+
+Use a vertical checklist with grouped phases, icons, explicit text states, and the current/failed phase expanded. Keep completed phases available for inspection. Avoid percentage complete or predicted finish times.
+
+| Action | Phase sequence and completion evidence |
+| --- | --- |
+| Create | Request/startup queue → prepare files and create deployment → start container → container readiness → game status → matching engine map. |
+| Start | Eligibility and restart capacity → start existing container → container and game verification. Do not imply files were redeployed. |
+| Restart | Eligibility, occupancy, and capacity → restart submission → container and game verification. Show any continuing recovery separately from request failure. |
+| Rebuild | Eligibility/capacity → remove old container and deployment → local files/registry cleanup → queue fresh instance → create/start → readiness and map verification. Preserve the reserved identity/port and show when the old instance is gone. |
+| Stop | Eligibility and occupancy → stop submission → explicit stopped-container confirmation. Explain that files and registration remain. |
+| Destroy | Eligibility and occupancy → container removal → deployment deletion → local files/registry cleanup. Retain the historical record after removal. |
+
+Rebuild grouping must retain each irreversible checkpoint. “Remote deployment deleted” and “Local cleanup complete” are different facts; fresh creation must never look complete while old files remain.
+
+Progress supports waiting, current, completed, failed, cancelled/not run, skipped when actually recorded, and unknown. Do not mark earlier steps complete simply because a string matched a later label. Legacy records retain their raw stage and known outcome, with unrecorded milestones labeled accordingly.
+
+The blocker panel shows the worker's actual reason: occupied/unknown players, missing configuration, requester hold, conflicting work, startup or restart capacity, removal/port dependency, container readiness, game evidence, or wrong map. Include the blocking operation/source link when known. Keep “Requested by” separate from “Waiting on.”
+
+Show relevant capacity limits, holders, and this operation's reservation from the same backend rules. If a submission rank is shown, label it as request order: blocked earlier jobs can be bypassed, and source priorities affect admission. Do not say “You are next.”
+
+Evidence rows show result, evaluation time, source, and expected/observed values where relevant. Engine-map verification does not independently verify a T6 variant/start location. Unknown players are never displayed as zero.
+
+Separate durable milestones from transient readiness checks. A passed container/game/map observation remains evidence of what passed at its timestamp; a later evaluation may make the current check failed or unknown. Do not leave stale green checks implying current readiness.
+
+### Actions and terminal states
+
+| Backend condition | Page behavior |
+| --- | --- |
+| `canCancelRemoval` | Offer **Cancel queued removal**, explaining that it is available only before removal begins. Preserve the existing atomic worker/cancellation race protection. |
+| `canResumeVerification` | Offer **Resume health verification** and explain that it observes the existing instance without creating or restarting it. |
+| `canResumeCleanup` | Offer **Resume cleanup**, including eligible deleted identities. Explain that Komodo deletion already succeeded. |
+| `canResumeRebuild` | Offer **Resume rebuild cleanup** and explain that local cleanup precedes creation in the reserved slot. |
+| Failed/interrupted without a resume capability | Lead with the recorded failure and relevant inspection destination. An uncertain remote response must not produce a generic retry command. |
+| Cancelled | State that the request is cancelled; distinguish recorded prior work from remaining work that did not run. Do not claim continued execution. |
+| Warning | Explain the specific remaining issue. Do not assume every warning concerns cleanup. |
+| Complete | Show the verified historical outcome and appropriate View server / Back to related work / Back to history link. |
+
+The API revalidates every action. A 409 triggers reconciliation of current capabilities and explains the state change; it never automatically repeats a mutation. Disable duplicate submissions while an action is pending, keep its accessible name, and preserve its result independently of background refresh. If an acknowledged successful action is followed by a failed refresh, retain its success acknowledgment alongside stale-data feedback; a refresh failure does not turn the action into a failure.
+
+“Launch another server” remains secondary beside the recipe, with its existing explanation that it creates a separate server using current promoted files. It is not a repair or replay of the recorded launch. Create and rebuild already save a recipe while queued, then recapture promoted files when creation starts. Label these **Setup captured when queued** versus **Files selected for this attempt**, with capture times and an explicit recipe status. The queued capture is not a version pin, and selected files do not prove installation succeeded. Legacy records without that distinction retain neutral recorded-recipe wording. Where no recipe exists, explain the absence.
+
+### Activity, refresh, and accessibility
+
+- Keep five recent events visible and show a targeted failure excerpt when needed. Retain full-history search, ordering, and Show more. Do not automatically open a large history when a failure arrives.
+- Load full history on demand in bounded pages rather than retransmitting the entire retained history every five seconds. Preserve the legacy API's default response through an opt-in page query.
+- Preserve search, disclosure, focus, and reading position across updates. While inspecting older history, show a new-event count and **Jump to latest** instead of moving the content underneath the reader.
+- Separate initial loading failure, background refresh failure, and action failure. Keep last-known detail visible with an explicit stale message and manual Refresh.
+- Show **Page updated** separately from **Last evaluated** and evidence receipt time. A successful GET does not prove the worker or server was checked recently. An unchanged event stream does not prove the worker is stalled.
+- Keep the existing five-second visibility-aware polling initially, including terminal records. Failed jobs may still have recovery activity; another tab may resume them; later deletion can change identity links and capabilities.
+- Use server-loaded initial detail. On route changes and actions, use abortable reads and request generations so old responses cannot replace new detail or feedback. Force reconciliation after an action even if a poll was already in progress.
+- Announce meaningful outcome/action changes politely, not every timer tick. Keep textual progress states and `aria-current`; errors remain near their actions. Use a stable loading layout and a clear missing-operation state.
+- Reuse existing primitives, spacing, colors, typography, and identity components. Verify long identifiers, keyboard access, visible focus, contrast, reduced motion, 320/390px layouts, tablet/desktop layouts, and 200% zoom.
+
+## Data and implementation design
+
+The page needs structured facts that the current response does not fully expose. Add a small typed read model and observation metadata; preserve the existing lifecycle state machine and execution predicates.
+
+| Contract area | Existing foundation | Required addition |
+| --- | --- | --- |
+| Identity and outcome | `server_deployments`, historical identity guards | Shared typed detail; recorded result remains distinct from current recovery. |
+| Phase and timing | Free-text `stage`, request/finish/submission times | Nullable stable phase key/start time; explicit checkpoint states/times; total elapsed remains based on the original request. |
+| Blocker | Worker decision branches and stage messages | Reason code, explanation, evaluated time, automatic continuation condition, and exact dependency reference where known. |
+| Command certainty | `submitted_at` is persisted before the network call | Distinguish submission attempted, reply accepted, uncertain outcome, and verified result. Never relabel submission intent as remote acceptance. |
+| Verification | Existing probes and transient results | Versioned snapshot with container/game/map/occupancy evidence, result/source/evaluation time, source receipt time where available, and expected/actual values. |
+| Recovery and capacity | `restart_rollout`, creation queue, saved concurrency | Exact operation-owned slot/reservation status, relevant limits/holders, and durable recovery/release evidence. |
+| Available actions | Existing `canResume*` and `canCancelRemoval` checks | Keep these authoritative; supply contextual explanation without creating new execution capabilities. |
+| Provenance and recipe | Sources, collection, saved launch recipe | Named/resolvable context links, missing/deleted-source handling, and explicit queued-capture versus attempted-selection status. |
+| Activity | Ordered event ID/time/message | Optional semantic kind/phase/details while preserving existing event text and order. |
+
+A suitable small schema extension is nullable `phase_key`, `phase_started_at`, `observed_at`, and versioned `observation_json` on the operation, plus optional event `kind`, `phase_key`, and allowlisted `details_json`. Final naming can follow repository conventions. Snapshot payloads remain bounded and contain only the fields needed by the page, not raw integration responses or credentials.
+
+Implementation rules:
+
+1. Record structured blockers and observations where the worker already makes decisions. Do not build a second eligibility engine in the API or browser. The detail GET remains read-only and performs no Komodo/RCON probes.
+2. Derive current capacity from existing scheduler helpers/tables, with an explicit snapshot time. Persist important milestones against the operation because server-owned recovery rows can later be replaced or deleted. Only attach a live slot when its `operation_id` matches; slot absence alone is not evidence of successful release.
+3. Update related state, observation, and semantic event atomically around each durable checkpoint, keeping network calls outside database transactions. Dual-write existing stage/event text, especially `REMOVING_SERVER`, which participates in recovery eligibility today.
+4. Record meaningful phase, blocker, command, and recovery transitions. Update observation timestamps on repeated checks without appending an event every poll. Changing player counts must not reset the phase's start time.
+5. Preserve prior failure and resume events when an existing operation resumes. New observations must not rewrite the earlier command result or borrow evidence from a newer instance. This does not introduce a new retry engine.
+6. For historical records, expose available durable facts and unknowns. Do not backfill invented success timestamps, evidence sources, or completed phases from ambiguous prose.
+7. Keep existing IDs, history, API properties, `/api/deployments` compatibility, and legacy redirects. Use additive migrations and test both fresh and upgraded databases. Preserve the legacy `deployment.warnings` string; expose a validated string array through the additive page projection, with an honest fallback for malformed historical values.
+
+Suggested boundaries:
+
+- Shared provisioning detail/progress types near [provisioning.ts](../src/lib/provisioning.ts); a pure presentation mapper for labels, action/state coverage, and legacy fallbacks.
+- Server serialization and persistence helpers near [history.ts](../src/lib/server/provisioning/history.ts), with instrumentation in [worker.ts](../src/lib/server/provisioning/worker.ts), [instances.ts](../src/lib/server/provisioning/instances.ts), and [restarts.ts](../src/lib/server/provisioning/restarts.ts).
+- An abortable provisioning refresh/action controller under `src/lib/ui/`, taking lessons from the existing metrics refresh controller without changing `pollVisible` globally.
+- A thin detail route composed from outcome/actions, execution/evidence, and supporting context. Make activity improvements opt-in or provisioning-specific so campaign/beta history defaults stay intact.
+- Reuse the already-fetched detail in [the server loader](../src/routes/provisioning/[id]/+page.server.ts), preserving its campaign fallback and 404 behavior.
+
+The current API returns all events on every poll; the existing Show more limits rendering only. Add an opt-in bounded detail query for this page and its initial server load: five recent entries, event total/latest ID, and the relevant failure excerpt. Fetch full history through `/api/provisioning/:id/events` in 25-entry pages with stable event-ID cursors, literal-text search, and explicit order. Keep existing default detail responses and `/api/deployments` behavior compatible. New events must not shift the active history page; show their count until the reader requests them. Test ordering and cursor stability with concurrent inserts, and measure representative long histories. This is a bounded query change, not a new event-storage system.
+
+## Discussion decisions
+
+| Proposal or disagreement | Decision and reason |
+| --- | --- |
+| Add a richer dashboard versus reorganize current capabilities | Reorganize around one operation and add only the evidence needed to explain it. Existing recipes, diagnostics, and recovery controls remain useful. |
+| Automatically expand all history on failure | Show the failure excerpt and five recent events; keep full expansion under operator control to avoid long, moving pages. |
+| Continue fetching all events because only five are displayed | Use an opt-in bounded detail/history API. A five-entry visual preview alone does not bound network or database work, and provisioning history is retained. Preserve legacy consumers. |
+| Compress rebuild into a few visual phases | Accepted only if remote deletion, local cleanup, replacement queueing, and verification remain separately inspectable. |
+| Improve progress using more stage-string matching | Rejected as the primary model. Use typed worker facts with an honest legacy fallback. Preserve old strings for compatibility and safety predicates. |
+| Show an exact queue position or ETA | Show actual blockers/capacity and optionally request order. Admission skips blocked requests; durations are not predictable from rank. |
+| Stop refreshing completed/failed jobs | Keep visibility-aware polling initially because recovery, deletion, and permitted actions can still change. |
+| Treat Deleted as making every action unavailable | Use backend capabilities. A historical server can still have permitted local cleanup without a live-server link. |
+| Treat failed request and recovery as one status | Keep them separate: the historical failure may remain while recovery continues and releases capacity. |
+| Add a new event system or scheduler | Rejected. Use additive snapshots and semantic metadata at existing execution boundaries. |
+
+## Delivery sequence
+
+These four increments form the proposed implementation scope. The data contract can be agreed first while layout work proceeds against fixtures.
+
+| Increment | Work | Reviewable result |
+| --- | --- | --- |
+| 1. Trustworthy operational facts | Shared types, additive migration, worker checkpoint/evidence writes, typed read model, capability and legacy handling. | All six actions provide accurate progress and unknowns without changing execution, command counts, occupancy rules, or slot release. |
+| 2. Correct state and refresh behavior | Replace string-derived presentation; fix cancelled/runtime and telemetry copy, deleted cleanup gating, historical outcomes; reuse server-loaded detail; isolate navigation/poll/action requests and feedback. | The existing page remains truthful during cancellation, resume, slow requests, offline refresh, navigation, and continued recovery after failure. |
+| 3. Complete detail-page composition | Outcome/action card, blocker/evidence panel, grouped vertical phases, five recent events, bounded full-history queries, metadata/provenance/recipe placement, copy links, return context, responsive/accessibility work. | Operators can find the job's outcome, blocking condition, and valid next action without searching the activity log. |
+| 4. Compatibility and release validation | Exercise the full scenario matrix, migration/API compatibility, shared-component regressions, browser behavior, docs, and long-history measurements. | Implementation is ready for normal deployment with documented semantics and evidence of preserved lifecycle guarantees. |
+
+Dependencies: evidence-dependent controls wait for increment 1; refresh and visual work can run in parallel against the agreed contract. Increment 4 is a release gate, not a later cleanup task.
+
+## Acceptance and verification
+
+The review baseline passed **84 tests across five suites**: provisioning, creation queue, operation history, deleted-server guards, and restart rollout. Those tests establish the existing baseline; they do not validate the proposed implementation. No full-build or accessibility-conformance claim is made for this planning work.
+
+Implementation acceptance:
+
+| Scenario | Required result |
+| --- | --- |
+| Every action across applicable queued/waiting/running/verifying/success/warning/failure/interruption/cancellation states | Correct headline, phase, evidence, timing, and permitted next action. Impossible combinations have explicit legacy/unknown handling. |
+| Occupied, unknown, stopped, stale, conflicting RCON/telemetry evidence | UI matches existing eligibility decisions; positive occupancy blocks disruption; stale/pre-submission/future telemetry cannot masquerade as accepted evidence. |
+| Capacity or parent/dependency hold | Actual blocker and matching links appear; no guaranteed admission order, fabricated check time, or ETA. |
+| Ambiguous remote reply, then recovery | No duplicate external command; slot stays reserved until the existing rules release/transfer it; historical failure and recovery evidence remain distinguishable. |
+| Partial destroy/rebuild and deleted identity | Confirmed remote deletion is retained; local cleanup can resume only when permitted; creation waits for full cleanup; old work never targets a replacement instance. |
+| Cancellation or resume racing a worker claim/newer operation | Backend remains authoritative; 409 reconciles the page; no automatic mutation replay or false success. |
+| Resume of the same operation ID | Prior failure outcome/time and explicit resume event remain preserved in semantic history while current progress/finish fields update; later completion records its own outcome without a new retry system. |
+| Telemetry success / wrong map / historical success | Evidence source and timestamps are accurate; expected and observed engine maps are clear; the page does not claim current health from old results. |
+| Slow poll + action, A→B→A navigation, stale error, unmount | Obsolete responses cannot affect current detail or action feedback; successful mutations invalidate stale reads and trigger reconciliation; pending state resets appropriately. If reconciliation fails, action acknowledgment and stale-data feedback remain distinct. |
+| Refresh failure and recovery | Last snapshot remains visible with stale labeling; Retry/Refresh works; action errors are not silently erased. |
+| Long activity, new events while reading | Search/order/disclosures/focus remain stable; new-event indicator works; no forced scrolling. Responses stay bounded; cursor pages have no duplicates/gaps within their documented ordering; legacy defaults remain compatible. Measure payload/query/render cost. |
+| Promotion occurs while create/rebuild is queued | Queued recipe is labeled as a capture, final attempted selection is recorded at creation, and the page does not imply pinned versions or successful installation. |
+| Legacy DB, absent recipe/evidence/source, deleted/reused name | Honest missing-data presentation, preserved IDs/events/foreign keys, safe links/capabilities, and unchanged legacy route/API compatibility. |
+| Keyboard, assistive announcements, phone, zoom, long text | No loss of reachable actions or page-wide overflow; progress is understandable without color; timers do not cause repeated announcements. |
+
+Extend behavior tests around these risks rather than snapshotting markup or mirroring DTO construction. Verify that detail GETs perform no external probes. Run the affected lifecycle/history/refresh tests, then the full suite, `npm run check`, and `npm run build` for the implementation. Validate desktop/mobile browser fixtures and update [Provisioning documentation](provisioning.md) and relevant README descriptions.
+
+Preserve the existing execution rules throughout: fresh occupancy gates, startup/restart limits, ambiguous-response handling, historical identity checks, exact cleanup checkpoints, requester ownership, and observation-only resumes. Normal deployment should sample both a newly instrumented operation and a legacy record; presentation rollback must not require deleting the new history.
+
+Outside this proposal: generic retry/force-run controls, new cancellation semantics, scheduler redesign, real-time transport migration, a general observability platform, broad redesigns of other operation pages, and predicted completion times.
@@ -75,6 +75,59 @@ new attempt, preserving prior history; paused or closed beta work cannot start a
 queued restart. Recovery of an already submitted restart continues even if its
 requester closes. Upgrades adopt existing restart intent for observation only.

+## Operation details
+
+Each job page puts its recorded outcome, blocking condition, and available next
+action before launch files. Execution is grouped into phases, with separate
+milestones for remote deletion, local cleanup, and replacement creation during a
+rebuild. Copy the operation ID or link when investigating or handing off a job.
+Related work links back to the requesting campaign, beta, maintenance item, or
+collection. A completed provisioning job does not imply that its requester has
+finished its own verification.
+
+New operations record structured verification evidence alongside their activity:
+the check result, source, evaluation time, and expected/observed map when relevant.
+Game status can be verified by RCON or accepted fresh telemetry. Engine-map checks
+do not independently verify a T6 start location. Passed checks describe their
+recorded observation; later checks can fail or become unknown. Completion describes
+what was verified at that time, while **View server** opens the current instance.
+Older history retains its recorded stages and outcomes; unavailable evidence is
+not inferred from stage wording.
+
+Submission attempted, command accepted, and verified completion are separate
+milestones. A failed restart may still have ongoing recovery observation and hold
+restart capacity. Its original failure remains in history even if recovery later
+succeeds and releases the slot. Capacity and request order are snapshots, not
+guarantees of the next job admitted or an estimated completion time. Blocked
+requests can be bypassed by eligible work.
+
+Recovery controls use the server's current eligibility checks. **Resume health
+verification** observes the existing instance without submitting another create or
+restart. **Resume cleanup** retries local cleanup after confirmed remote deletion,
+including eligible historical deleted identities; replacement or newer-operation
+guards still apply. **Resume rebuild cleanup** completes cleanup before creating
+the replacement in its reserved slot. Resuming the same operation preserves its
+earlier failure and resume events. Uncertain commands have no generic Retry action.
+
+Create and rebuild capture a setup when queued and select the latest promoted
+files again when creation begins. The page distinguishes **Setup captured when
+queued** from **Files selected for this attempt**. A queued capture does not pin
+versions; an attempted selection is not proof that every file was installed.
+**Launch another server** uses the latest promoted setup with a fresh identity and
+port, rather than repairing or replaying the recorded job.
+
+Five recent events remain visible. Expand full history to search, change order,
+and load older pages. A fixed history watermark keeps the viewed pages stable as
+events arrive; **Jump to latest** incorporates new activity. The detail view
+fetches a bounded recent-event window, while full history loads on demand.
+
+The page refreshes every five seconds while visible, including terminal jobs whose
+recovery or available actions may still change. **Page updated** describes the
+last successful detail fetch; it is separate from the worker's evaluation and
+telemetry receipt times. A refresh failure keeps the last snapshot visible and
+does not erase an acknowledged action result. These timestamps are not worker
+heartbeat or completion-time promises.
+
 ## Routes and compatibility

 - `/provisioning`: active operations and history filtered by action.
@@ -89,6 +142,16 @@ requester closes. Upgrades adopt existing restart intent for observation only.
 - `/api/provisioning/:id`, `/:id/resume`, `/:id/cancel`, `/preview`, and
  `/house-party` preserve the existing operation, recovery, removal cancellation,
  launch preview, and fleet destruction APIs.
+- `GET /api/provisioning/:id?events=recent` returns the same detail with five recent
+  events plus activity totals/latest ID and a failure excerpt. Omitting the option
+  preserves the full legacy event response. Progress, capacity, named related work,
+  and normalized warning messages are additive fields; the original warning string
+  and lifecycle fields remain available.
+- `GET /api/provisioning/:id/events` reads history with `order=newest|oldest`,
+  literal-text `query`, `limit` (default 25, maximum 100), an exclusive event-ID
+  `cursor`, and an upper-ID `watermark`. Keep the returned watermark when loading
+  more and use `nextCursor` until it is null. Begin without a watermark to include
+  newly arrived events. These reads do not probe Komodo or RCON.

 Old `/deployments` pages redirect to their `/provisioning` equivalents, preserving
 query parameters. Collection launch links at `/provisioning/collections/:id` or