electric-cherry/docs/guides/provisioning.md

Provisioning

Open Activity → Activity log for retained lifecycle, host, campaign, and maintenance history. Use Manage servers from that page or command search for server queues, bulk cancellation, and lifecycle actions. Fleet → Collections owns reusable lineups and collection launch progress; their individual operations remain visible in Provisioning. Provisioning owns the queues, safety checks, execution, and recovery for all six actions. Create a server using Create server. For an existing server, use its visible Restart server or Start server action, or open More actions. Only actions supported by current host and container evidence are shown. Confirmations name the server and its current players and round; disruptive actions still wait for a fresh empty-lobby check. Manage server on the operations page offers the same actions for a selected server.

Provisioning badges show the recorded lifecycle state consistently on the list and job details: Queued, Waiting, Running, Validating, Completed, Completed with warnings, Failed, Interrupted, or Cancelled. Validating is the display name for awaiting-health. The list's Status column shows only the state badge. Open Details or the individual job to inspect execution progress and waiting reasons. File preparation can occur while a job is Queued or Waiting; Running begins when the host command is submitted and includes waiting for its acknowledgment. These progress details do not change the badge, status counts, or filters.

The Active tab shows Validating jobs first, then Running, Waiting, and Queued. Within each state, older requests appear first, with recording order breaking timestamp ties. History shows the newest requests first. Needs attention filters unresolved Failed, Interrupted, and Completed with warnings outcomes; their individual state badges remain visible. This display order does not change execution priority or capacity admission.

Create server opens one dialog with game and map choices followed by automatic readiness checks. Blockers explain what needs attention before launch. Runtime and network overrides are under Advanced settings; image, configuration, managed files, upload time, and internal server ID are under Setup details. The footer keeps Cancel and Launch server available while reviewing the setup. Create and launch reusable lineups separately in Fleet → Collections.

Collection launches prepare their fleet preview automatically and show fill and exact modes together. Review changes and game key coverage, then use the persistent footer to queue work. Capacity estimates are advisory; jobs can wait for hosts and resources. Exact mode covers both games and requires confirmation of its explicit removal list, which resets whenever the preview is refreshed. See Collections for the review details and safe submission retries when a response is lost.

Action	Behavior
Create	Create a new instance using the latest promoted files and verify container health, game status, the selected map, and the Plutonium revision.
Start	Start the existing container and verify container health and RCON.
Restart	Wait until empty, restart the existing container, and verify container health and RCON.
Rebuild	Wait until empty, replace the instance through the host agent, preserving its server identity and port while advancing its generation. Verify container health, game status, map, and Plutonium revision.
Stop	Wait until empty, stop the container, and verify that the host agent reports it stopped. Keep the instance for a later Start.
Remove	Wait until empty, remove the container and its files through the host agent, then retire the instance and registration.

Safety and verification

Beta ownership and rebuilds

Servers linked to a running or ready-to-promote beta cannot be removed or rebuilt. Promote or roll back the beta first, including paused or failed test plans. Fleet removal and exact collection launches queue nothing if any server selected for removal belongs to an active beta. Provisioning checks again before teardown; already queued removals wait while their server belongs to an active beta. Opening a beta also requires cancelling or finishing any pending removal or rebuild on its selected servers.

Rebuild requires confirmation because no instance data is preserved. Its fresh files come from the latest promoted setup when creation begins, including changes promoted while waiting. Its assigned key in Keys must exist before teardown and stays reserved throughout the rebuild. Blueprint definitions, promoted artifacts, saved keys, and historical activity belong to Electric Cherry, and are not instance files.

Rebuild reserves the original ID and port throughout teardown and creation, even when a lower lobby is available. It shares the existing creation queue and startup limit. The agent records each operation durably, and the controller reconciles its receipt after reconnecting or restarting. An interrupted fresh startup can resume health verification without repeating the operation. Inspect failures before making another request.

Health and runtime checks

If the host agent explicitly reports an unhealthy container during Create or Rebuild verification, that operation forces one restart, then forces one rebuild if the container remains unhealthy. Recovery preserves the selected launch files, credentials, server identity, and port. If both attempts leave it unhealthy, the operation fails for inspection instead of repeating recovery. Starting, restarting, and unknown container states continue waiting for readiness. This requires host agent 1.8.1 or newer.

Create and Rebuild finish only after RCON reports a runtime revision at least as new as the current Plutonium production release. The worker checks the production manifest and records the runtime revision for fleet and server version status as part of provisioning. An unavailable release or unknown runtime revision keeps the job awaiting verification and holds its startup slot while the worker retries. A confirmed older runtime fails the operation with its expected and observed revisions; it does not trigger an automatic restart. Check Plutainer's startup updater configuration and server logs before retrying. Telemetry can verify game status and map, but cannot verify the Plutonium revision.

Start and Restart share the rolling restart limit with campaigns, betas, and maintenance. A successful agent receipt is followed by actual container and game-status checks. Failed or ambiguous restart requests retain their recovery slot and are not automatically replayed. Stop requires an explicit stopped container state; an unreachable host agent is not evidence that a stop succeeded.

Game key pools

Each create reserves one unused Plutonium key for its game before the launch is queued. T5 and T6 have independent pools; any map or lobby within that game can use a free key. Stopping a server keeps the reservation, and a rebuild keeps the same key. Confirmed removal releases it for another launch. Failed or ambiguous creation keeps the assignment while cleanup may still be required.

An unstarted creation cancelled before dispatch releases its key immediately. Administrators can add and remove available keys in Admin → Keys; assigned keys cannot be removed. Existing keys are migrated into their game pool, retaining assignments for existing servers and pending work. If a legacy rebuild has an uncertain result and used a different key, both possible keys remain reserved until that server is removed. Labels are for identification only and do not change the hostname registered with Plutonium.

Player checks and administrator overrides

Disruptive operations require a fresh zero-player result from RCON or accepted telemetry, unless the host agent positively reports a stopped container. Fresh occupied telemetry vetoes a conflicting empty RCON reading. Reports must be received within 90 seconds; recovery verification also excludes reports from before the submitted operation. Telemetry can verify the game's activity and selected map after the host agent reports healthy. Runtime actions wait for conflicting campaign, beta, maintenance, or provisioning work. Duplicate pending requests for the same action return the existing operation; a different action returns a conflict. Requests and events survive application restarts.

Rebuild and recovery within Create or Rebuild treat an explicitly unhealthy container as empty without requiring a player count. The agent checks health again before disruption; if the container has recovered, normal player checks apply. This exception does not apply to unknown health or other lifecycle actions.

For a failed container stuck waiting for player status, an administrator can open its queued or waiting Restart, Stop, Rebuild, or Destroy operation and choose Admin override…. The confirmation modal explains the interruption and deletion risks. Confirming bypasses player-count and empty-server checks in both the controller and host agent for that operation only. The approval persists across restarts and records the administrator and time in the operation history. The host agent must support overrides (version 1.4.15 or newer); older agents show an update requirement. Submitted commands, host migrations, and completed operations cannot be overridden. Capacity, conflicting work, instance identity, file validation, and post-action health checks still apply.

Campaigns, betas and maintenance

Campaign and beta file changes submit durable rebuild requests to Provisioning and observe the resulting operation by ID. Their pages link to that operation, and its history links back to the requesting work. Provisioning owns the empty-server check, conflict handling, capacity limits, remote command, and health verification. The callers retain file deployment, beta evidence, and maintenance outcome verification. Jobs remain visible while waiting for players, other work, or integration configuration.

File rebuilds require host agent 1.9.2 or later. They capture rollback snapshots before replacing the instance, and install the requested files over the promoted setup. Queued file activations wait for the agent update when necessary. Maintenance jobs and beta recovery cycles without file changes still use restarts.

Maintenance also submits recovery operations and links each attempt to its incident. A failed Maintenance restart can hand its reserved slot to a rebuild after five minutes; unrelated recovery slots remain held. Drift campaigns follow their linked rebuild without requesting another restart. Their file evidence and history remain available after the instance is removed.

Each campaign activation has its own durable operation and file selection. A failed or interrupted request keeps its original ID and never automatically replays. An explicit beta retry starts a new attempt, preserving prior history; paused or closed beta work cannot start a queued restart. Recovery of an already submitted restart continues even if its requester closes. Upgrades adopt existing restart intent for observation only.

Operation details

Each job page puts its recorded outcome, blocking condition, and available next action before launch files. Execution is grouped into phases, with separate milestones for remote deletion, local cleanup, and replacement creation during a rebuild. Copy the operation ID or link when investigating or handing off a job. Related work links back to the requesting campaign, beta, maintenance item, or collection. A completed provisioning job does not imply that its requester has finished its own verification.

Optional timestamps appear only when recorded. Launch files appears for Create and Rebuild, or when an older operation has a recorded recipe. Start, Restart, and Stop use the existing instance without selecting new launch files.

While a job is active, Controller activity lists current work and elapsed time. Background scans run independently of provisioning.

Verification evidence

New operations record structured verification evidence alongside their activity: the check result, source, evaluation time, and expected/observed map or Plutonium revision when relevant. Create and Rebuild show Verify Plutonium revision, with separate evidence for the production manifest and the running revision from RCON. Game status can be verified by RCON or accepted fresh telemetry. Engine-map checks do not independently verify a T6 start location. Passed checks describe their recorded observation; later checks can fail or become unknown. Completion describes what was verified at that time, while View server opens the current instance. Older history retains its recorded stages and outcomes; unavailable evidence is not inferred from stage wording.

Submission attempted, command accepted, and verified completion are separate milestones. A failed restart may still have ongoing recovery observation and hold restart capacity. Its original failure remains in history even if recovery later succeeds and releases the slot. Capacity and request order are snapshots, not guarantees of the next job admitted or an estimated completion time. Blocked requests can be bypassed by eligible work.

Recovery actions and launch files

Recovery controls use the server's current eligibility checks. Resume health verification observes the existing instance without submitting another create or restart. Resume cleanup retries local cleanup after confirmed remote deletion, including eligible historical deleted identities; replacement or newer-operation guards still apply. Resume rebuild cleanup completes cleanup before creating the replacement in its reserved slot. Resuming the same operation preserves its earlier failure and resume events. Uncertain commands have no generic Retry action.

Cancelling an operation does not stop a command already delivered to the host. The confirmation explains that remaining work and health checks stop, while changes already made remain. Cancel creation and remove server also deletes the launch's container and instance files, even if players have joined or their status is unknown. Shared game files are kept, and the key returns to its game pool after confirmed removal.

A later rebuild waits for that command to settle. Its waiting message includes the host command's state and last reported error; View cancelled operation opens the original command details. If an interrupted rebuild removed its old container before creating the replacement, the host can confirm failure after verifying that both containers are absent and the instance identity is unchanged. The original diagnostic remains available, and the queued rebuild can continue. Uncertain inspections keep it waiting; the old command is never replayed merely because its receipt was lost.

Create and rebuild capture a setup when queued and select the latest promoted files again when creation begins. The Launch files card shows one selection status beside the capture time: Captured when queued, Selected when creation began, or Recorded setup; selection time unknown for older records. A queued capture does not pin versions; an attempted selection is not proof that every file was installed. Expand the managed file list to inspect versions, paths, checksums, and the recipe fingerprint.

History and refresh behavior

Five recent events remain visible. Expand full history to search, change order, and load older pages. A fixed history watermark keeps the viewed pages stable as events arrive; Jump to latest incorporates new activity. The detail view fetches a bounded recent-event window, while full history loads on demand.

The page refreshes every five seconds while visible, including terminal jobs whose recovery or available actions may still change. Page updated describes the last successful detail fetch; it is separate from the worker's evaluation and telemetry receipt times. A refresh failure keeps the last snapshot visible and does not erase an acknowledged action result. These timestamps are not worker heartbeat or completion-time promises.

Routes and compatibility

• /provisioning: active operations and history filtered by action.
• /provisioning/:id: operation stages, errors, timestamps, and activity.
• /collections: saved lineups, active launches, and launch history.
• /collections/launches/:id: collection progress, with links to each operation.
• GET /api/provisioning: lifecycle history (deployments response property).
• POST /api/provisioning: queue { "server": "t5_zombie_theater_1", "action": "start" }. Supported actions are start, restart, rebuild, and stop. Returns HTTP 202 with deploymentId. Create continues through POST /api/servers; Destroy uses DELETE /api/servers/:name.
• /api/provisioning/:id, /:id/resume, /:id/cancel, /preview, and /house-party preserve the existing operation, recovery, removal cancellation, launch preview, and fleet removal APIs.
• POST /api/provisioning/:id/cancel cancels any active create, start, restart, rebuild, stop, or destroy job, including health verification. It releases that job's recovery capacity and prevents further steps; completed effects are not undone. Undelivered host commands are withdrawn. Commands already delivered retain their receipts, and replacements wait for those mutations to settle. Cancelling is idempotent. Lists and details expose canCancel.
• POST /api/provisioning/:id/cancel-creation retains the explicit native creation cleanup action: cancel the launch and remove its partial instance. Cancelling a job alone retains its existing instance/files for inspection or later cleanup.
• Beta rollback cancels conflicting provisioning jobs on its original servers before queuing independent restoration restarts. It does not wait for the superseded restart's health verification. Native activation receipts must still provide durable before images; rollback reconciles completed receipts itself. Manual rollback keeps the replacement jobs held until the campaign is resumed.
• GET /api/provisioning/:id?events=recent returns the same detail with five recent events plus activity totals/latest ID and a failure excerpt. Omitting the option preserves the full legacy event response. Progress, capacity, named related work, and normalized warning messages are additive fields; the original warning string and lifecycle fields remain available.
• GET /api/provisioning/:id/events reads history with order=newest|oldest, literal-text query, limit (default 25, maximum 100), an exclusive event-ID cursor, and an upper-ID watermark. Keep the returned watermark when loading more and use nextCursor until it is null. Begin without a watermark to include newly arrived events. These reads do not contact host agents or probe RCON.

Old /deployments pages redirect to their /provisioning equivalents, preserving query parameters. Collection launch links at /provisioning/collections/:id or /deployments/collections/:id redirect to /collections/launches/:id and preserve query parameters. /api/deployments remains compatible, including old campaign detail endpoints. No history or IDs are renamed or discarded.

Start and Stop use durable instance.start and instance.stop host-agent operations. Rebuild uses instance.rebuild and replaces the instance files. See native host management for command delivery and recovery.

Concurrency limits

Admin → Settings groups Concurrency separately from history and storage settings, with one Save changes action for all pending edits. Existing fleet restart, startup and removal limits continue to apply. Additional per-host restart, startup and removal limits, a combined lifecycle limit, a rebuild limit, and a limit on hosts preparing game files default to 0 (no additional cap), preserving existing throughput on upgrade.

Lifecycle work counts once toward the combined limit, including launch preparation and unresolved delivered commands. Rebuilds include drain migrations and retain capacity between source removal and destination creation. Per-host recovery slots remain occupied until health verification. Lowering limits does not cancel work. Game-file preparation is limited when commands are delivered; downloads and imports share the cap, including manual requests. Replayed or interrupted commands retain the same host slot. Runtime image pulls and per-host file transfer counts are unchanged.