192 lines
13 KiB
Markdown
192 lines
13 KiB
Markdown
# Provisioning
|
|
|
|
Open **Operations → Provisioning** for instance lifecycle work and persistent
|
|
activity. **Operations → Collections** owns reusable lineups and collection launch
|
|
progress; their individual operations remain visible in Provisioning.
|
|
Provisioning owns the queues, safety checks, execution, and
|
|
recovery for all six actions. Create a server using **Create server**. For an existing
|
|
server, choose **Manage server** or open its **Server actions** menu.
|
|
|
|
| Action | Behavior |
|
|
| --- | --- |
|
|
| Create | Create a new instance using the latest promoted files and verify container health, game status, the selected map, and the Plutonium revision. |
|
|
| Start | Start the existing container and verify container health and RCON. |
|
|
| Restart | Wait until empty, restart the existing container, and verify container health and RCON. |
|
|
| Rebuild | Wait until empty, destroy the instance, delete all server files and its registration, then create a fresh instance in the same server slot and port. Verify container health, game status, map, and Plutonium revision. |
|
|
| Stop | Wait until empty, stop the container, and verify that Komodo reports it stopped. Keep the instance for a later Start. |
|
|
| Destroy | Wait until empty, remove the container and Komodo deployment, then delete server files and registration. |
|
|
|
|
Servers linked to a running or ready-to-promote beta cannot be destroyed or
|
|
rebuilt. Promote or roll back the beta first, including paused or failed test
|
|
plans. Fleet removal and exact collection launches queue nothing if any server
|
|
selected for removal belongs to an active beta. Provisioning checks again before
|
|
teardown; already queued removals wait while their server belongs to an active
|
|
beta. Opening a beta also requires cancelling or finishing any pending destruction
|
|
or rebuild on its selected servers.
|
|
|
|
Rebuild requires confirmation because **no instance data is preserved**. Its fresh
|
|
files come from the latest promoted setup when creation begins, including changes
|
|
promoted while waiting. The independently saved key in **Keys** must exist before
|
|
teardown. Blueprint definitions, promoted artifacts, saved keys, and historical
|
|
activity belong to Electric Cherry, and are not instance files.
|
|
|
|
Rebuild reserves the original ID and port throughout teardown and creation, even
|
|
when a lower lobby is available. It shares the existing creation queue and startup
|
|
limit. A failed removal never proceeds to creation. Once Komodo deletion is
|
|
confirmed, failed local cleanup offers **Resume rebuild cleanup**; creation cannot
|
|
proceed until the old files are fully removed. An interrupted fresh startup can
|
|
resume health verification, without repeating teardown or creation. Inspect other
|
|
failures before making another request.
|
|
|
|
Create and Rebuild finish only after RCON reports a runtime revision at least as
|
|
new as the current Plutonium production release. The worker checks the production
|
|
manifest and records the runtime revision for fleet and server version status as
|
|
part of provisioning. An unavailable release or unknown runtime revision keeps the
|
|
job awaiting verification and holds its startup slot while the worker retries.
|
|
A confirmed older runtime fails the operation with its expected and observed
|
|
revisions; it does not trigger an automatic restart. Check Plutainer's startup
|
|
updater configuration and server logs before retrying. Telemetry can verify game
|
|
status and map, but cannot verify the Plutonium revision.
|
|
|
|
Start and Restart share the rolling restart limit with campaigns, betas, and
|
|
maintenance. A successful API reply is followed by a recovery delay and actual
|
|
container and game-status checks. Failed or ambiguous restart requests retain their recovery
|
|
slot and are not automatically replayed. Stop requires an explicit stopped container state; an
|
|
unreachable Komodo API is not evidence that a stop succeeded.
|
|
|
|
Disruptive operations require a fresh zero-player result from RCON or accepted
|
|
telemetry, unless Komodo positively reports a stopped container. Fresh occupied
|
|
telemetry vetoes a conflicting empty RCON reading. Reports must be received within
|
|
90 seconds; recovery verification also excludes reports from before the submitted
|
|
operation. Telemetry can verify the game's activity and selected map after Komodo
|
|
reports healthy. Runtime actions wait for conflicting
|
|
campaign, beta, maintenance, or provisioning work. Duplicate pending requests for
|
|
the same action return the existing operation; a different action returns a
|
|
conflict. Requests and events survive application restarts.
|
|
|
|
Campaigns, beta runs, and maintenance submit durable restart requests to
|
|
Provisioning and observe the resulting operation by ID. Their pages link to that
|
|
operation, and its history links back to the requesting work. Provisioning owns
|
|
the empty-server check, conflict handling, restart capacity, remote command, and
|
|
health verification. The callers retain file deployment, beta evidence, and
|
|
maintenance outcome verification. Jobs remain visible while waiting for players,
|
|
other work, or integration configuration.
|
|
|
|
[Maintenance](maintenance.md) also submits recovery operations and links each attempt
|
|
to its incident. A failed Maintenance restart can hand its reserved slot to a rebuild
|
|
after five minutes; unrelated recovery slots remain held. Drift campaigns follow
|
|
their linked rebuild without requesting another restart. Their file evidence and
|
|
history remain available after the instance is removed.
|
|
|
|
Campaign changes queued before submission can share one restart. Changes arriving
|
|
after submission require a new operation. A failed or interrupted request keeps
|
|
its original ID and never automatically replays. An explicit beta retry starts a
|
|
new attempt, preserving prior history; paused or closed beta work cannot start a
|
|
queued restart. Recovery of an already submitted restart continues even if its
|
|
requester closes. Upgrades adopt existing restart intent for observation only.
|
|
|
|
## Operation details
|
|
|
|
Each job page puts its recorded outcome, blocking condition, and available next
|
|
action before launch files. Execution is grouped into phases, with separate
|
|
milestones for remote deletion, local cleanup, and replacement creation during a
|
|
rebuild. Copy the operation ID or link when investigating or handing off a job.
|
|
Related work links back to the requesting campaign, beta, maintenance item, or
|
|
collection. A completed provisioning job does not imply that its requester has
|
|
finished its own verification.
|
|
|
|
New operations record structured verification evidence alongside their activity:
|
|
the check result, source, evaluation time, and expected/observed map or Plutonium
|
|
revision when relevant. Create and Rebuild show **Verify Plutonium revision**, with
|
|
separate evidence for the production manifest and the running revision from RCON.
|
|
Game status can be verified by RCON or accepted fresh telemetry. Engine-map checks
|
|
do not independently verify a T6 start location. Passed checks describe their
|
|
recorded observation; later checks can fail or become unknown. Completion describes
|
|
what was verified at that time, while **View server** opens the current instance.
|
|
Older history retains its recorded stages and outcomes; unavailable evidence is
|
|
not inferred from stage wording.
|
|
|
|
Submission attempted, command accepted, and verified completion are separate
|
|
milestones. A failed restart may still have ongoing recovery observation and hold
|
|
restart capacity. Its original failure remains in history even if recovery later
|
|
succeeds and releases the slot. Capacity and request order are snapshots, not
|
|
guarantees of the next job admitted or an estimated completion time. Blocked
|
|
requests can be bypassed by eligible work.
|
|
|
|
Recovery controls use the server's current eligibility checks. **Resume health
|
|
verification** observes the existing instance without submitting another create or
|
|
restart. **Resume cleanup** retries local cleanup after confirmed remote deletion,
|
|
including eligible historical deleted identities; replacement or newer-operation
|
|
guards still apply. **Resume rebuild cleanup** completes cleanup before creating
|
|
the replacement in its reserved slot. Resuming the same operation preserves its
|
|
earlier failure and resume events. Uncertain commands have no generic Retry action.
|
|
|
|
Create and rebuild capture a setup when queued and select the latest promoted
|
|
files again when creation begins. The page distinguishes **Setup captured when
|
|
queued** from **Files selected for this attempt**. A queued capture does not pin
|
|
versions; an attempted selection is not proof that every file was installed.
|
|
**Launch another server** uses the latest promoted setup with a fresh identity and
|
|
port, rather than repairing or replaying the recorded job.
|
|
|
|
Five recent events remain visible. Expand full history to search, change order,
|
|
and load older pages. A fixed history watermark keeps the viewed pages stable as
|
|
events arrive; **Jump to latest** incorporates new activity. The detail view
|
|
fetches a bounded recent-event window, while full history loads on demand.
|
|
|
|
The page refreshes every five seconds while visible, including terminal jobs whose
|
|
recovery or available actions may still change. **Page updated** describes the
|
|
last successful detail fetch; it is separate from the worker's evaluation and
|
|
telemetry receipt times. A refresh failure keeps the last snapshot visible and
|
|
does not erase an acknowledged action result. These timestamps are not worker
|
|
heartbeat or completion-time promises.
|
|
|
|
## Routes and compatibility
|
|
|
|
- `/provisioning`: active operations and history filtered by action.
|
|
- `/provisioning/:id`: operation stages, errors, timestamps, and activity.
|
|
- `/collections`: saved lineups, active launches, and launch history.
|
|
- `/collections/launches/:id`: collection progress, with links to each operation.
|
|
- `GET /api/provisioning`: lifecycle history (`deployments` response property).
|
|
- `POST /api/provisioning`: queue `{ "server": "t5_zombie_theater_1", "action": "start" }`.
|
|
Supported actions are `start`, `restart`, `rebuild`, and `stop`. Returns HTTP 202
|
|
with `deploymentId`. Create continues through `POST /api/servers`; Destroy uses
|
|
`DELETE /api/servers/:name`.
|
|
- `/api/provisioning/:id`, `/:id/resume`, `/:id/cancel`, `/preview`, and
|
|
`/house-party` preserve the existing operation, recovery, removal cancellation,
|
|
launch preview, and fleet destruction APIs.
|
|
- `POST /api/provisioning/:id/cancel` cancels any active create, start, restart,
|
|
rebuild, stop, or destroy job, including health verification. It releases that
|
|
job's recovery capacity and prevents further steps; completed effects are not
|
|
undone. Undelivered host commands are withdrawn. Commands already delivered
|
|
retain their receipts, and replacements wait for those mutations to settle.
|
|
Cancelling is idempotent. Lists and details expose `canCancel`.
|
|
- `POST /api/provisioning/:id/cancel-creation` retains the explicit native creation
|
|
cleanup action: cancel the launch and remove its partial instance. Cancelling
|
|
a job alone retains its existing instance/files for inspection or later cleanup.
|
|
- Beta rollback cancels conflicting provisioning jobs on its original servers
|
|
before queuing independent restoration restarts. It does not wait for the
|
|
superseded restart's health verification. Native activation receipts must still
|
|
provide durable before images; rollback reconciles completed receipts itself.
|
|
Manual rollback keeps the replacement jobs held until the campaign is resumed.
|
|
- `GET /api/provisioning/:id?events=recent` returns the same detail with five recent
|
|
events plus activity totals/latest ID and a failure excerpt. Omitting the option
|
|
preserves the full legacy event response. Progress, capacity, named related work,
|
|
and normalized warning messages are additive fields; the original warning string
|
|
and lifecycle fields remain available.
|
|
- `GET /api/provisioning/:id/events` reads history with `order=newest|oldest`,
|
|
literal-text `query`, `limit` (default 25, maximum 100), an exclusive event-ID
|
|
`cursor`, and an upper-ID `watermark`. Keep the returned watermark when loading
|
|
more and use `nextCursor` until it is null. Begin without a watermark to include
|
|
newly arrived events. These reads do not probe Komodo or RCON.
|
|
|
|
Old `/deployments` pages redirect to their `/provisioning` equivalents, preserving
|
|
query parameters. Collection launch links at `/provisioning/collections/:id` or
|
|
`/deployments/collections/:id` redirect to `/collections/launches/:id` and preserve
|
|
query parameters. `/api/deployments` remains compatible, including old campaign
|
|
detail endpoints. No history or IDs are renamed or discarded.
|
|
|
|
Start and Stop use Komodo's corresponding runtime commands. Rebuild uses full
|
|
destruction and creation, not Komodo's redeploy command. See the
|
|
[Komodo execute API](https://docs.rs/komodo_client/latest/komodo_client/api/execute/index.html)
|
|
for the underlying container operations.
|