electric-cherry/docs/provisioning-improvement-plan.md

# Provisioning job detail improvement plan

Review date: September 28, 2026. Source baseline: `5b65520`.

The recommendation is to make `/provisioning/:id` a complete operational record: identify the job, explain its current outcome and blocker, expose the evidence behind that outcome, and show the next permitted action. Progress and recovery should precede historical launch files.

Approved and implemented on September 28, 2026. The original review and agreed scope remain below; see the [implementation and validation record](provisioning-implementation.md) for delivered behavior, checks, and updated screenshots.

## Review method and participants

The requested panel covered three UI/UX roles, two CI/CD systems engineering roles, and one software engineering role. The platform allowed only three child-agent threads and rejected additional spawns even after those reviews completed. The three threads therefore performed two separate rounds, first UX and then engineering. This provides six role-based reviews, rather than six independent agents. Each round included direct peer discussion; engineering reviewed and challenged the UX proposals.

| Review role | Proposal and contribution |
| --- | --- |
| UI/UX: information architecture | Prioritize job identity, outcome, blocker, safe action, progress, activity, and broader work context. Separate historical completion from current server health. |
| UI/UX: interaction and accessibility | Use state-specific recovery actions, stable refresh behavior, persistent action feedback, keyboard-accessible progress, and usable history while updates arrive. |
| UI/UX: visual design | Replace the overloaded summary and wrapping step chips with a compact outcome card, grouped vertical progress, and supporting metadata. Reuse the existing design system. |
| CI/CD systems: lifecycle and queues | Preserve destructive checkpoints, occupancy gates, scheduling, command certainty, identity guards, and recovery-slot semantics. Make queue claims match actual admission behavior. |
| CI/CD systems: observability and reliability | Record the source and age of verification evidence, distinguish page freshness from worker checks, preserve recovery evidence, and validate compatibility and failure scenarios. |
| Software engineering | Define shared types and component boundaries, reuse server-loaded detail, fix route/poll/action races, and stage additive implementation with regression coverage. |

The review covered the page, shared history component, lifecycle APIs, worker, queues, persistence, related collection/maintenance/campaign pages, and relevant tests. Root also inspected the current source in Firefox using an isolated temporary database, disabled integration credentials, and synthetic detail responses. Desktop captures used 1440px width; mobile used 390px. These are composed UI scenarios, not executions of real provisioning workflows or measurements of production use.

Review assets: [create waiting for a map](provisioning-review-assets/create-map-wait-desktop.png), [mobile rebuild composition](provisioning-review-assets/rebuild-mobile.png), and [deleted identity with resumable cleanup](provisioning-review-assets/deleted-cleanup-desktop.png).

## What feels incomplete today

The page already has useful recovery buttons, server diagnostics links, a recorded launch recipe, source backlinks, elapsed time, progress steps, and searchable activity. Preserve those capabilities. The gaps concern their priority and the reliability of the story they tell.

| Finding | Effect on the operator | Evidence |
| --- | --- | --- |
| One summary mixes status, actions, errors, warnings, diagnostics, map, timing, and provenance. The recipe appears before progress. | Monitoring and recovery compete with secondary information. | [Detail page](../src/routes/provisioning/[id]/+page.svelte), summary and recipe markup. In the synthetic desktop create case, the flow began about 1,073px down. |
| Rebuild renders 12 equally weighted wrapping step chips. | Removal, cleanup, and fresh creation do not read as distinct phases. | [Detail page](../src/routes/provisioning/[id]/+page.svelte), `steps`; [OperationHistory](../src/lib/ui/components/OperationHistory.svelte). |
| Progress and step duration depend on matching human-readable stage/event text. | New wording can move the highlighted step backward; changes in player count can appear to restart a step timer. | Detail page `currentStep` and `stageStarted`; [creation verification](../src/lib/server/provisioning/instances.ts) now emits “RCON or fresh telemetry” stages. |
| Runtime cancellation is handled after runtime-specific explanation logic. | A cancelled start/restart/stop can still claim automatic verification is underway. | Detail page `explanation`. |
| The deleted-server flag suppresses every cleanup button and the active-state presentation. | A permitted local cleanup can become inaccessible or appear inactive after resuming. | Detail page recovery controls and `active` versus [history.ts](../src/lib/server/provisioning/history.ts), `canResumeCleanup`. Missing button reproduced with a synthetic permitted response. |
| Completion copy asserts RCON health in the present tense. | Telemetry-based verification is misreported, and historical success can look like current health. | Detail page explanations; [status-evidence.ts](../src/lib/server/fleet/status-evidence.ts), `probeServerStatus`. |
| One refresh guard and one error string handle reads, mutations, and navigation. | A post-action reload can be dropped, an old request can affect another job, and polling can erase an action error. | Detail page `load`, `resume`, `cancelRemoval`; source-inferred race cases. |
| A failed restart can still hold a recovery slot. | A simple failed badge does not explain continued observation or effects on other jobs. | [restarts.ts](../src/lib/server/provisioning/restarts.ts), `refreshRestartRollout`. |
| Activity is collapsed, operation ID is absent from the page body, and related-work links are scattered. | Investigation and handoff require avoidable searching and navigation. | Detail page and shared history component. |

## Proposed page

Use one reading order across all six actions. At wide widths, metadata and launch inputs can occupy a supporting column. On phones they follow progress and activity in the DOM. Avoid a sticky sidebar.

```text
Provisioning / Restart server
Origins · Lobby 1     t6_zombie_origins_1
Operation ID [Copy]   [Copy link]                  [View server]

[Waiting] Waiting for players to leave
Last evaluated: 3 players · source and observation time
Continues automatically after an eligible empty/stopped check.
Relevant permitted action or dependency link
Total elapsed · Time in phase · Page updated / Refresh

EXECUTION AND EVIDENCE                     OPERATION CONTEXT
  Eligibility                             Requested by / requested at
  Container action                        Submission attempted / finished
  Readiness verification                  Related campaign/beta/maintenance
    Container                             Collection launch
    Game status
    Selected map, when required            Recorded launch files
                                          [Launch another server]
RECENT ACTIVITY
  Five latest entries + relevant failure
  [Full searchable history]
```

The example evidence requires the new contract below; do not populate it from guessed values.

### Identity, outcome, and next action

- Show the action, friendly map/lobby title, technical server name, and copyable operation ID/link. Use existing identity and copy components.
- Lead with the specific blocker or outcome, followed by what happens next. Place the recorded error and permitted recovery control together. Give warnings a labeled explanation rather than color alone.
- Treat deletion as historical identity context. It must not replace the outcome, hide an actionable cleanup, or link an old operation to a replacement server with the same name.
- Determine active progress from the operation and recovery state, independently of whether a live server link is available.
- Distinguish a recorded outcome from ongoing recovery. A failed request can show “Recovery is still being checked; restart capacity remains reserved” when the backend records that condition.
- Use past-tense completion language with its timestamp: “Creation completed. Container readiness, game status, and the selected engine map were verified at …”. Name RCON or telemetry only when recorded. Current health remains on the server page.
- Preserve an explicit return link to the originating Provisioning filters or parent work, with a safe local fallback. Group all requesting campaigns, betas, maintenance records, and collection launches under Related work. Keep retained source IDs visible when their originating record no longer exists, without a broken link.
- Finishing a restart does not mean its campaign, beta, or maintenance verification has finished. Link back to that work with accurate wording.

### Progress and evidence

Use a vertical checklist with grouped phases, icons, explicit text states, and the current/failed phase expanded. Keep completed phases available for inspection. Avoid percentage complete or predicted finish times.

| Action | Phase sequence and completion evidence |
| --- | --- |
| Create | Request/startup queue → prepare files and create deployment → start container → container readiness → game status → matching engine map. |
| Start | Eligibility and restart capacity → start existing container → container and game verification. Do not imply files were redeployed. |
| Restart | Eligibility, occupancy, and capacity → restart submission → container and game verification. Show any continuing recovery separately from request failure. |
| Rebuild | Eligibility/capacity → remove old container and deployment → local files/registry cleanup → queue fresh instance → create/start → readiness and map verification. Preserve the reserved identity/port and show when the old instance is gone. |
| Stop | Eligibility and occupancy → stop submission → explicit stopped-container confirmation. Explain that files and registration remain. |
| Destroy | Eligibility and occupancy → container removal → deployment deletion → local files/registry cleanup. Retain the historical record after removal. |

Rebuild grouping must retain each irreversible checkpoint. “Remote deployment deleted” and “Local cleanup complete” are different facts; fresh creation must never look complete while old files remain.

Progress supports waiting, current, completed, failed, cancelled/not run, skipped when actually recorded, and unknown. Do not mark earlier steps complete simply because a string matched a later label. Legacy records retain their raw stage and known outcome, with unrecorded milestones labeled accordingly.

The blocker panel shows the worker's actual reason: occupied/unknown players, missing configuration, requester hold, conflicting work, startup or restart capacity, removal/port dependency, container readiness, game evidence, or wrong map. Include the blocking operation/source link when known. Keep “Requested by” separate from “Waiting on.”

Show relevant capacity limits, holders, and this operation's reservation from the same backend rules. If a submission rank is shown, label it as request order: blocked earlier jobs can be bypassed, and source priorities affect admission. Do not say “You are next.”

Evidence rows show result, evaluation time, source, and expected/observed values where relevant. Engine-map verification does not independently verify a T6 variant/start location. Unknown players are never displayed as zero.

Separate durable milestones from transient readiness checks. A passed container/game/map observation remains evidence of what passed at its timestamp; a later evaluation may make the current check failed or unknown. Do not leave stale green checks implying current readiness.

### Actions and terminal states

| Backend condition | Page behavior |
| --- | --- |
| `canCancelRemoval` | Offer **Cancel queued removal**, explaining that it is available only before removal begins. Preserve the existing atomic worker/cancellation race protection. |
| `canResumeVerification` | Offer **Resume health verification** and explain that it observes the existing instance without creating or restarting it. |
| `canResumeCleanup` | Offer **Resume cleanup**, including eligible deleted identities. Explain that Komodo deletion already succeeded. |
| `canResumeRebuild` | Offer **Resume rebuild cleanup** and explain that local cleanup precedes creation in the reserved slot. |
| Failed/interrupted without a resume capability | Lead with the recorded failure and relevant inspection destination. An uncertain remote response must not produce a generic retry command. |
| Cancelled | State that the request is cancelled; distinguish recorded prior work from remaining work that did not run. Do not claim continued execution. |
| Warning | Explain the specific remaining issue. Do not assume every warning concerns cleanup. |
| Complete | Show the verified historical outcome and appropriate View server / Back to related work / Back to history link. |

The API revalidates every action. A 409 triggers reconciliation of current capabilities and explains the state change; it never automatically repeats a mutation. Disable duplicate submissions while an action is pending, keep its accessible name, and preserve its result independently of background refresh. If an acknowledged successful action is followed by a failed refresh, retain its success acknowledgment alongside stale-data feedback; a refresh failure does not turn the action into a failure.

“Launch another server” remains secondary beside the recipe, with its existing explanation that it creates a separate server using current promoted files. It is not a repair or replay of the recorded launch. Create and rebuild already save a recipe while queued, then recapture promoted files when creation starts. Label these **Setup captured when queued** versus **Files selected for this attempt**, with capture times and an explicit recipe status. The queued capture is not a version pin, and selected files do not prove installation succeeded. Legacy records without that distinction retain neutral recorded-recipe wording. Where no recipe exists, explain the absence.

### Activity, refresh, and accessibility

- Keep five recent events visible and show a targeted failure excerpt when needed. Retain full-history search, ordering, and Show more. Do not automatically open a large history when a failure arrives.
- Load full history on demand in bounded pages rather than retransmitting the entire retained history every five seconds. Preserve the legacy API's default response through an opt-in page query.
- Preserve search, disclosure, focus, and reading position across updates. While inspecting older history, show a new-event count and **Jump to latest** instead of moving the content underneath the reader.
- Separate initial loading failure, background refresh failure, and action failure. Keep last-known detail visible with an explicit stale message and manual Refresh.
- Show **Page updated** separately from **Last evaluated** and evidence receipt time. A successful GET does not prove the worker or server was checked recently. An unchanged event stream does not prove the worker is stalled.
- Keep the existing five-second visibility-aware polling initially, including terminal records. Failed jobs may still have recovery activity; another tab may resume them; later deletion can change identity links and capabilities.
- Use server-loaded initial detail. On route changes and actions, use abortable reads and request generations so old responses cannot replace new detail or feedback. Force reconciliation after an action even if a poll was already in progress.
- Announce meaningful outcome/action changes politely, not every timer tick. Keep textual progress states and `aria-current`; errors remain near their actions. Use a stable loading layout and a clear missing-operation state.
- Reuse existing primitives, spacing, colors, typography, and identity components. Verify long identifiers, keyboard access, visible focus, contrast, reduced motion, 320/390px layouts, tablet/desktop layouts, and 200% zoom.

## Data and implementation design

The page needs structured facts that the current response does not fully expose. Add a small typed read model and observation metadata; preserve the existing lifecycle state machine and execution predicates.

| Contract area | Existing foundation | Required addition |
| --- | --- | --- |
| Identity and outcome | `server_deployments`, historical identity guards | Shared typed detail; recorded result remains distinct from current recovery. |
| Phase and timing | Free-text `stage`, request/finish/submission times | Nullable stable phase key/start time; explicit checkpoint states/times; total elapsed remains based on the original request. |
| Blocker | Worker decision branches and stage messages | Reason code, explanation, evaluated time, automatic continuation condition, and exact dependency reference where known. |
| Command certainty | `submitted_at` is persisted before the network call | Distinguish submission attempted, reply accepted, uncertain outcome, and verified result. Never relabel submission intent as remote acceptance. |
| Verification | Existing probes and transient results | Versioned snapshot with container/game/map/occupancy evidence, result/source/evaluation time, source receipt time where available, and expected/actual values. |
| Recovery and capacity | `restart_rollout`, creation queue, saved concurrency | Exact operation-owned slot/reservation status, relevant limits/holders, and durable recovery/release evidence. |
| Available actions | Existing `canResume*` and `canCancelRemoval` checks | Keep these authoritative; supply contextual explanation without creating new execution capabilities. |
| Provenance and recipe | Sources, collection, saved launch recipe | Named/resolvable context links, missing/deleted-source handling, and explicit queued-capture versus attempted-selection status. |
| Activity | Ordered event ID/time/message | Optional semantic kind/phase/details while preserving existing event text and order. |

A suitable small schema extension is nullable `phase_key`, `phase_started_at`, `observed_at`, and versioned `observation_json` on the operation, plus optional event `kind`, `phase_key`, and allowlisted `details_json`. Final naming can follow repository conventions. Snapshot payloads remain bounded and contain only the fields needed by the page, not raw integration responses or credentials.

Implementation rules:

1. Record structured blockers and observations where the worker already makes decisions. Do not build a second eligibility engine in the API or browser. The detail GET remains read-only and performs no Komodo/RCON probes.
2. Derive current capacity from existing scheduler helpers/tables, with an explicit snapshot time. Persist important milestones against the operation because server-owned recovery rows can later be replaced or deleted. Only attach a live slot when its `operation_id` matches; slot absence alone is not evidence of successful release.
3. Update related state, observation, and semantic event atomically around each durable checkpoint, keeping network calls outside database transactions. Dual-write existing stage/event text, especially `REMOVING_SERVER`, which participates in recovery eligibility today.
4. Record meaningful phase, blocker, command, and recovery transitions. Update observation timestamps on repeated checks without appending an event every poll. Changing player counts must not reset the phase's start time.
5. Preserve prior failure and resume events when an existing operation resumes. New observations must not rewrite the earlier command result or borrow evidence from a newer instance. This does not introduce a new retry engine.
6. For historical records, expose available durable facts and unknowns. Do not backfill invented success timestamps, evidence sources, or completed phases from ambiguous prose.
7. Keep existing IDs, history, API properties, `/api/deployments` compatibility, and legacy redirects. Use additive migrations and test both fresh and upgraded databases. Preserve the legacy `deployment.warnings` string; expose a validated string array through the additive page projection, with an honest fallback for malformed historical values.

Suggested boundaries:

- Shared provisioning detail/progress types near [provisioning.ts](../src/lib/provisioning.ts); a pure presentation mapper for labels, action/state coverage, and legacy fallbacks.
- Server serialization and persistence helpers near [history.ts](../src/lib/server/provisioning/history.ts), with instrumentation in [worker.ts](../src/lib/server/provisioning/worker.ts), [instances.ts](../src/lib/server/provisioning/instances.ts), and [restarts.ts](../src/lib/server/provisioning/restarts.ts).
- An abortable provisioning refresh/action controller under `src/lib/ui/`, taking lessons from the existing metrics refresh controller without changing `pollVisible` globally.
- A thin detail route composed from outcome/actions, execution/evidence, and supporting context. Make activity improvements opt-in or provisioning-specific so campaign/beta history defaults stay intact.
- Reuse the already-fetched detail in [the server loader](../src/routes/provisioning/[id]/+page.server.ts), preserving its campaign fallback and 404 behavior.

The current API returns all events on every poll; the existing Show more limits rendering only. Add an opt-in bounded detail query for this page and its initial server load: five recent entries, event total/latest ID, and the relevant failure excerpt. Fetch full history through `/api/provisioning/:id/events` in 25-entry pages with stable event-ID cursors, literal-text search, and explicit order. Keep existing default detail responses and `/api/deployments` behavior compatible. New events must not shift the active history page; show their count until the reader requests them. Test ordering and cursor stability with concurrent inserts, and measure representative long histories. This is a bounded query change, not a new event-storage system.

## Discussion decisions

| Proposal or disagreement | Decision and reason |
| --- | --- |
| Add a richer dashboard versus reorganize current capabilities | Reorganize around one operation and add only the evidence needed to explain it. Existing recipes, diagnostics, and recovery controls remain useful. |
| Automatically expand all history on failure | Show the failure excerpt and five recent events; keep full expansion under operator control to avoid long, moving pages. |
| Continue fetching all events because only five are displayed | Use an opt-in bounded detail/history API. A five-entry visual preview alone does not bound network or database work, and provisioning history is retained. Preserve legacy consumers. |
| Compress rebuild into a few visual phases | Accepted only if remote deletion, local cleanup, replacement queueing, and verification remain separately inspectable. |
| Improve progress using more stage-string matching | Rejected as the primary model. Use typed worker facts with an honest legacy fallback. Preserve old strings for compatibility and safety predicates. |
| Show an exact queue position or ETA | Show actual blockers/capacity and optionally request order. Admission skips blocked requests; durations are not predictable from rank. |
| Stop refreshing completed/failed jobs | Keep visibility-aware polling initially because recovery, deletion, and permitted actions can still change. |
| Treat Deleted as making every action unavailable | Use backend capabilities. A historical server can still have permitted local cleanup without a live-server link. |
| Treat failed request and recovery as one status | Keep them separate: the historical failure may remain while recovery continues and releases capacity. |
| Add a new event system or scheduler | Rejected. Use additive snapshots and semantic metadata at existing execution boundaries. |

## Delivery sequence

These four increments form the proposed implementation scope. The data contract can be agreed first while layout work proceeds against fixtures.

| Increment | Work | Reviewable result |
| --- | --- | --- |
| 1. Trustworthy operational facts | Shared types, additive migration, worker checkpoint/evidence writes, typed read model, capability and legacy handling. | All six actions provide accurate progress and unknowns without changing execution, command counts, occupancy rules, or slot release. |
| 2. Correct state and refresh behavior | Replace string-derived presentation; fix cancelled/runtime and telemetry copy, deleted cleanup gating, historical outcomes; reuse server-loaded detail; isolate navigation/poll/action requests and feedback. | The existing page remains truthful during cancellation, resume, slow requests, offline refresh, navigation, and continued recovery after failure. |
| 3. Complete detail-page composition | Outcome/action card, blocker/evidence panel, grouped vertical phases, five recent events, bounded full-history queries, metadata/provenance/recipe placement, copy links, return context, responsive/accessibility work. | Operators can find the job's outcome, blocking condition, and valid next action without searching the activity log. |
| 4. Compatibility and release validation | Exercise the full scenario matrix, migration/API compatibility, shared-component regressions, browser behavior, docs, and long-history measurements. | Implementation is ready for normal deployment with documented semantics and evidence of preserved lifecycle guarantees. |

Dependencies: evidence-dependent controls wait for increment 1; refresh and visual work can run in parallel against the agreed contract. Increment 4 is a release gate, not a later cleanup task.

## Acceptance and verification

The review baseline passed **84 tests across five suites**: provisioning, creation queue, operation history, deleted-server guards, and restart rollout. Those tests establish the existing baseline; they do not validate the proposed implementation. No full-build or accessibility-conformance claim is made for this planning work.

Implementation acceptance:

| Scenario | Required result |
| --- | --- |
| Every action across applicable queued/waiting/running/verifying/success/warning/failure/interruption/cancellation states | Correct headline, phase, evidence, timing, and permitted next action. Impossible combinations have explicit legacy/unknown handling. |
| Occupied, unknown, stopped, stale, conflicting RCON/telemetry evidence | UI matches existing eligibility decisions; positive occupancy blocks disruption; stale/pre-submission/future telemetry cannot masquerade as accepted evidence. |
| Capacity or parent/dependency hold | Actual blocker and matching links appear; no guaranteed admission order, fabricated check time, or ETA. |
| Ambiguous remote reply, then recovery | No duplicate external command; slot stays reserved until the existing rules release/transfer it; historical failure and recovery evidence remain distinguishable. |
| Partial destroy/rebuild and deleted identity | Confirmed remote deletion is retained; local cleanup can resume only when permitted; creation waits for full cleanup; old work never targets a replacement instance. |
| Cancellation or resume racing a worker claim/newer operation | Backend remains authoritative; 409 reconciles the page; no automatic mutation replay or false success. |
| Resume of the same operation ID | Prior failure outcome/time and explicit resume event remain preserved in semantic history while current progress/finish fields update; later completion records its own outcome without a new retry system. |
| Telemetry success / wrong map / historical success | Evidence source and timestamps are accurate; expected and observed engine maps are clear; the page does not claim current health from old results. |
| Slow poll + action, A→B→A navigation, stale error, unmount | Obsolete responses cannot affect current detail or action feedback; successful mutations invalidate stale reads and trigger reconciliation; pending state resets appropriately. If reconciliation fails, action acknowledgment and stale-data feedback remain distinct. |
| Refresh failure and recovery | Last snapshot remains visible with stale labeling; Retry/Refresh works; action errors are not silently erased. |
| Long activity, new events while reading | Search/order/disclosures/focus remain stable; new-event indicator works; no forced scrolling. Responses stay bounded; cursor pages have no duplicates/gaps within their documented ordering; legacy defaults remain compatible. Measure payload/query/render cost. |
| Promotion occurs while create/rebuild is queued | Queued recipe is labeled as a capture, final attempted selection is recorded at creation, and the page does not imply pinned versions or successful installation. |
| Legacy DB, absent recipe/evidence/source, deleted/reused name | Honest missing-data presentation, preserved IDs/events/foreign keys, safe links/capabilities, and unchanged legacy route/API compatibility. |
| Keyboard, assistive announcements, phone, zoom, long text | No loss of reachable actions or page-wide overflow; progress is understandable without color; timers do not cause repeated announcements. |

Extend behavior tests around these risks rather than snapshotting markup or mirroring DTO construction. Verify that detail GETs perform no external probes. Run the affected lifecycle/history/refresh tests, then the full suite, `npm run check`, and `npm run build` for the implementation. Validate desktop/mobile browser fixtures and update [Provisioning documentation](provisioning.md) and relevant README descriptions.

Preserve the existing execution rules throughout: fresh occupancy gates, startup/restart limits, ambiguous-response handling, historical identity checks, exact cleanup checkpoints, requester ownership, and observation-only resumes. Normal deployment should sample both a newly instrumented operation and a legacy record; presentation rollback must not require deleting the new history.

Outside this proposal: generic retry/force-run controls, new cancellation semantics, scheduler redesign, real-time transport migration, a general observability platform, broad redesigns of other operation pages, and predicted completion times.