electric-cherry/docs/guides/provisioning.md

# Provisioning

Open **Operations → Provisioning** for instance lifecycle work and persistent
activity. **Fleet → Collections** owns reusable lineups and collection launch
progress; their individual operations remain visible in Provisioning.
Provisioning owns the queues, safety checks, execution, and
recovery for all six actions. Create a server using **Create server**. For an existing
server, choose **Manage server** or open its **Server actions** menu.

**Create server** opens one dialog with game and map choices followed by automatic
readiness checks. Blockers explain what needs attention before launch. Runtime
and network overrides are under **Advanced settings**; image, configuration,
managed files, upload time, and internal server ID are under **Setup details**.
The footer keeps **Cancel** and **Launch server** available while reviewing the
setup. Create and launch reusable lineups separately in **Fleet → Collections**.

Collection launches prepare their fleet preview automatically and show fill and
exact modes together. Review changes and game key coverage, then use the
persistent footer to queue work. Capacity estimates are advisory; jobs can wait
for hosts and resources. Exact mode covers both games and requires confirmation
of its explicit removal list, which resets whenever the preview is refreshed.
See [Collections](collections.md) for the review details and safe submission
retries when a response is lost.

| Action | Behavior |
| --- | --- |
| Create | Create a new instance using the latest promoted files and verify container health, game status, the selected map, and the Plutonium revision. |
| Start | Start the existing container and verify container health and RCON. |
| Restart | Wait until empty, restart the existing container, and verify container health and RCON. |
| Rebuild | Wait until empty, replace the instance through the host agent, preserving its server identity and port while advancing its generation. Verify container health, game status, map, and Plutonium revision. |
| Stop | Wait until empty, stop the container, and verify that the host agent reports it stopped. Keep the instance for a later Start. |
| Destroy | Wait until empty, remove the container and its files through the host agent, then retire the instance and registration. |

## Safety and verification

### Beta ownership and rebuilds

Servers linked to a running or ready-to-promote beta cannot be destroyed or
rebuilt. Promote or roll back the beta first, including paused or failed test
plans. Fleet removal and exact collection launches queue nothing if any server
selected for removal belongs to an active beta. Provisioning checks again before
teardown; already queued removals wait while their server belongs to an active
beta. Opening a beta also requires cancelling or finishing any pending destruction
or rebuild on its selected servers.

Rebuild requires confirmation because **no instance data is preserved**. Its fresh
files come from the latest promoted setup when creation begins, including changes
promoted while waiting. Its assigned key in **Keys** must exist before teardown
and stays reserved throughout the rebuild. Blueprint definitions, promoted
artifacts, saved keys, and historical activity belong to Electric Cherry, and are
not instance files.

Rebuild reserves the original ID and port throughout teardown and creation, even
when a lower lobby is available. It shares the existing creation queue and startup
limit. The agent records each operation durably, and the controller reconciles its
receipt after reconnecting or restarting. An interrupted fresh startup can resume
health verification without repeating the operation. Inspect failures before
making another request.

### Health and runtime checks

Create and Rebuild finish only after RCON reports a runtime revision at least as
new as the current Plutonium production release. The worker checks the production
manifest and records the runtime revision for fleet and server version status as
part of provisioning. An unavailable release or unknown runtime revision keeps the
job awaiting verification and holds its startup slot while the worker retries.
A confirmed older runtime fails the operation with its expected and observed
revisions; it does not trigger an automatic restart. Check Plutainer's startup
updater configuration and server logs before retrying. Telemetry can verify game
status and map, but cannot verify the Plutonium revision.

Start and Restart share the rolling restart limit with campaigns, betas, and
maintenance. A successful agent receipt is followed by actual container and game-status checks. Failed or ambiguous restart requests retain their recovery
slot and are not automatically replayed. Stop requires an explicit stopped container state; an
unreachable host agent is not evidence that a stop succeeded.

### Game key pools

Each create reserves one unused Plutonium key for its game before the launch is
queued. T5 and T6 have independent pools; any map or lobby within that game can
use a free key. Stopping a server keeps the reservation, and a rebuild keeps the
same key. Confirmed removal releases it for another launch. Failed or ambiguous
creation keeps the assignment while cleanup may still be required.

An unstarted creation cancelled before dispatch releases its key immediately.
Administrators can add and remove available keys in **Administration → Keys**;
assigned keys cannot be removed. Existing keys are migrated into their game
pool, retaining assignments for existing servers and pending work. If a legacy
rebuild has an uncertain result and used a different key, both possible keys
remain reserved until that server is destroyed. Labels are for identification
only and do not change the hostname registered with Plutonium.

### Player checks and administrator overrides

Disruptive operations require a fresh zero-player result from RCON or accepted
telemetry, unless the host agent positively reports a stopped container. Fresh occupied
telemetry vetoes a conflicting empty RCON reading. Reports must be received within
90 seconds; recovery verification also excludes reports from before the submitted
operation. Telemetry can verify the game's activity and selected map after the host agent
reports healthy. Runtime actions wait for conflicting
campaign, beta, maintenance, or provisioning work. Duplicate pending requests for
the same action return the existing operation; a different action returns a
conflict. Requests and events survive application restarts.

For a failed container stuck waiting for player status, an administrator can open
its queued or waiting **Restart**, **Stop**, **Rebuild**, or **Destroy** operation
and choose **Admin override…**. The confirmation modal explains the interruption
and deletion risks. Confirming bypasses player-count and empty-server checks in
both the controller and host agent for that operation only. The approval persists
across restarts and records the administrator and time in the operation history.
The host agent must support overrides (version 1.4.15 or newer); older agents show
an update requirement. Submitted commands, host migrations, and completed operations
cannot be overridden. Capacity, conflicting work, instance identity, file validation,
and post-action health checks still apply.

## Campaigns, betas and maintenance

Campaigns, beta runs, and maintenance submit durable restart requests to
Provisioning and observe the resulting operation by ID. Their pages link to that
operation, and its history links back to the requesting work. Provisioning owns
the empty-server check, conflict handling, restart capacity, remote command, and
health verification. The callers retain file deployment, beta evidence, and
maintenance outcome verification. Jobs remain visible while waiting for players,
other work, or integration configuration.

[Maintenance](maintenance.md) also submits recovery operations and links each attempt
to its incident. A failed Maintenance restart can hand its reserved slot to a rebuild
after five minutes; unrelated recovery slots remain held. Drift campaigns follow
their linked rebuild without requesting another restart. Their file evidence and
history remain available after the instance is removed.

Each campaign activation has its own durable operation and file selection. A failed or interrupted request keeps
its original ID and never automatically replays. An explicit beta retry starts a
new attempt, preserving prior history; paused or closed beta work cannot start a
queued restart. Recovery of an already submitted restart continues even if its
requester closes. Upgrades adopt existing restart intent for observation only.

## Operation details

Each job page puts its recorded outcome, blocking condition, and available next
action before launch files. Execution is grouped into phases, with separate
milestones for remote deletion, local cleanup, and replacement creation during a
rebuild. Copy the operation ID or link when investigating or handing off a job.
Related work links back to the requesting campaign, beta, maintenance item, or
collection. A completed provisioning job does not imply that its requester has
finished its own verification.

### Verification evidence

New operations record structured verification evidence alongside their activity:
the check result, source, evaluation time, and expected/observed map or Plutonium
revision when relevant. Create and Rebuild show **Verify Plutonium revision**, with
separate evidence for the production manifest and the running revision from RCON.
Game status can be verified by RCON or accepted fresh telemetry. Engine-map checks
do not independently verify a T6 start location. Passed checks describe their
recorded observation; later checks can fail or become unknown. Completion describes
what was verified at that time, while **View server** opens the current instance.
Older history retains its recorded stages and outcomes; unavailable evidence is
not inferred from stage wording.

Submission attempted, command accepted, and verified completion are separate
milestones. A failed restart may still have ongoing recovery observation and hold
restart capacity. Its original failure remains in history even if recovery later
succeeds and releases the slot. Capacity and request order are snapshots, not
guarantees of the next job admitted or an estimated completion time. Blocked
requests can be bypassed by eligible work.

### Recovery actions and launch files

Recovery controls use the server's current eligibility checks. **Resume health
verification** observes the existing instance without submitting another create or
restart. **Resume cleanup** retries local cleanup after confirmed remote deletion,
including eligible historical deleted identities; replacement or newer-operation
guards still apply. **Resume rebuild cleanup** completes cleanup before creating
the replacement in its reserved slot. Resuming the same operation preserves its
earlier failure and resume events. Uncertain commands have no generic Retry action.

Create and rebuild capture a setup when queued and select the latest promoted
files again when creation begins. The page distinguishes **Setup captured when
queued** from **Files selected for this attempt**. A queued capture does not pin
versions; an attempted selection is not proof that every file was installed.
**Launch another server** opens the creation dialog with the recorded game and
map as editable starting values. It checks the latest promoted setup and creates
a separate server with a fresh identity and port.

### History and refresh behavior

Five recent events remain visible. Expand full history to search, change order,
and load older pages. A fixed history watermark keeps the viewed pages stable as
events arrive; **Jump to latest** incorporates new activity. The detail view
fetches a bounded recent-event window, while full history loads on demand.

The page refreshes every five seconds while visible, including terminal jobs whose
recovery or available actions may still change. **Page updated** describes the
last successful detail fetch; it is separate from the worker's evaluation and
telemetry receipt times. A refresh failure keeps the last snapshot visible and
does not erase an acknowledged action result. These timestamps are not worker
heartbeat or completion-time promises.

## Routes and compatibility

- `/provisioning`: active operations and history filtered by action.
- `/provisioning/:id`: operation stages, errors, timestamps, and activity.
- `/collections`: saved lineups, active launches, and launch history.
- `/collections/launches/:id`: collection progress, with links to each operation.
- `GET /api/provisioning`: lifecycle history (`deployments` response property).
- `POST /api/provisioning`: queue `{ "server": "t5_zombie_theater_1", "action": "start" }`.
  Supported actions are `start`, `restart`, `rebuild`, and `stop`. Returns HTTP 202
  with `deploymentId`. Create continues through `POST /api/servers`; Destroy uses
  `DELETE /api/servers/:name`.
- `/api/provisioning/:id`, `/:id/resume`, `/:id/cancel`, `/preview`, and
  `/house-party` preserve the existing operation, recovery, removal cancellation,
  launch preview, and fleet destruction APIs.
- `POST /api/provisioning/:id/cancel` cancels any active create, start, restart,
  rebuild, stop, or destroy job, including health verification. It releases that
  job's recovery capacity and prevents further steps; completed effects are not
  undone. Undelivered host commands are withdrawn. Commands already delivered
  retain their receipts, and replacements wait for those mutations to settle.
  Cancelling is idempotent. Lists and details expose `canCancel`.
- `POST /api/provisioning/:id/cancel-creation` retains the explicit native creation
  cleanup action: cancel the launch and remove its partial instance. Cancelling
  a job alone retains its existing instance/files for inspection or later cleanup.
- Beta rollback cancels conflicting provisioning jobs on its original servers
  before queuing independent restoration restarts. It does not wait for the
  superseded restart's health verification. Native activation receipts must still
  provide durable before images; rollback reconciles completed receipts itself.
  Manual rollback keeps the replacement jobs held until the campaign is resumed.
- `GET /api/provisioning/:id?events=recent` returns the same detail with five recent
  events plus activity totals/latest ID and a failure excerpt. Omitting the option
  preserves the full legacy event response. Progress, capacity, named related work,
  and normalized warning messages are additive fields; the original warning string
  and lifecycle fields remain available.
- `GET /api/provisioning/:id/events` reads history with `order=newest|oldest`,
  literal-text `query`, `limit` (default 25, maximum 100), an exclusive event-ID
  `cursor`, and an upper-ID `watermark`. Keep the returned watermark when loading
  more and use `nextCursor` until it is null. Begin without a watermark to include
  newly arrived events. These reads do not contact host agents or probe RCON.

Old `/deployments` pages redirect to their `/provisioning` equivalents, preserving
query parameters. Collection launch links at `/provisioning/collections/:id` or
`/deployments/collections/:id` redirect to `/collections/launches/:id` and preserve
query parameters. `/api/deployments` remains compatible, including old campaign
detail endpoints. No history or IDs are renamed or discarded.

Start and Stop use durable `instance.start` and `instance.stop` host-agent
operations. Rebuild uses `instance.rebuild` and replaces the instance files.
See [native host management](native-hosts.md) for command delivery and recovery.

## Concurrency limits

Administration → Settings separates Concurrency and Retention, with independent
save actions. Existing fleet restart, startup and destruction limits continue to
apply. Additional per-host restart, startup and destruction limits, a combined
lifecycle limit, a rebuild limit, and a limit on hosts preparing game files default
to 0 (no additional cap), preserving existing throughput on upgrade.

Lifecycle work counts once toward the combined limit, including launch preparation
and unresolved delivered commands. Rebuilds include drain migrations and retain
capacity between source removal and destination creation. Per-host recovery slots
remain occupied until health verification. Lowering limits does not cancel work.
Game-file preparation is limited when commands are delivered; downloads and imports
share the cap, including manual requests. Replayed or interrupted commands retain
the same host slot. Runtime image pulls and per-host file transfer counts are unchanged.