electric-cherry/docs/provisioning.md

Provisioning

Open Operations → Provisioning for instance lifecycle work and persistent activity. Operations → Collections owns reusable lineups and collection launch progress; their individual operations remain visible in Provisioning. Provisioning owns the queues, safety checks, execution, and recovery for all six actions. Create a server using Create server. For an existing server, choose Manage server or open its Server actions menu.

Action	Behavior
Create	Create a new instance using the latest promoted files and verify container health, game status, the selected map, and the Plutonium revision.
Start	Start the existing container and verify container health and RCON.
Restart	Wait until empty, restart the existing container, and verify container health and RCON.
Rebuild	Wait until empty, destroy the instance, delete all server files and its registration, then create a fresh instance in the same server slot and port. Verify container health, game status, map, and Plutonium revision.
Stop	Wait until empty, stop the container, and verify that Komodo reports it stopped. Keep the instance for a later Start.
Destroy	Wait until empty, remove the container and Komodo deployment, then delete server files and registration.

Servers linked to a running or ready-to-promote beta cannot be destroyed or rebuilt. Promote or roll back the beta first, including paused or failed test plans. Fleet removal and exact collection launches queue nothing if any server selected for removal belongs to an active beta. Provisioning checks again before teardown; already queued removals wait while their server belongs to an active beta. Opening a beta also requires cancelling or finishing any pending destruction or rebuild on its selected servers.

Rebuild requires confirmation because no instance data is preserved. Its fresh files come from the latest promoted setup when creation begins, including changes promoted while waiting. The independently saved key in Keys must exist before teardown. Blueprint definitions, promoted artifacts, saved keys, and historical activity belong to Electric Cherry, and are not instance files.

Rebuild reserves the original ID and port throughout teardown and creation, even when a lower lobby is available. It shares the existing creation queue and startup limit. A failed removal never proceeds to creation. Once Komodo deletion is confirmed, failed local cleanup offers Resume rebuild cleanup; creation cannot proceed until the old files are fully removed. An interrupted fresh startup can resume health verification, without repeating teardown or creation. Inspect other failures before making another request.

Create and Rebuild finish only after RCON reports a runtime revision at least as new as the current Plutonium production release. The worker checks the production manifest and records the runtime revision for fleet and server version status as part of provisioning. An unavailable release or unknown runtime revision keeps the job awaiting verification and holds its startup slot while the worker retries. A confirmed older runtime fails the operation with its expected and observed revisions; it does not trigger an automatic restart. Check Plutainer's startup updater configuration and server logs before retrying. Telemetry can verify game status and map, but cannot verify the Plutonium revision.

Start and Restart share the rolling restart limit with campaigns, betas, and maintenance. A successful API reply is followed by a recovery delay and actual container and game-status checks. Failed or ambiguous restart requests retain their recovery slot and are not automatically replayed. Stop requires an explicit stopped container state; an unreachable Komodo API is not evidence that a stop succeeded.

Disruptive operations require a fresh zero-player result from RCON or accepted telemetry, unless Komodo positively reports a stopped container. Fresh occupied telemetry vetoes a conflicting empty RCON reading. Reports must be received within 45 seconds; recovery verification also excludes reports from before the submitted operation. Telemetry can verify the game's activity and selected map after Komodo reports healthy. Runtime actions wait for conflicting campaign, beta, maintenance, or provisioning work. Duplicate pending requests for the same action return the existing operation; a different action returns a conflict. Requests and events survive application restarts.

Campaigns, beta runs, and maintenance submit durable restart requests to Provisioning and observe the resulting operation by ID. Their pages link to that operation, and its history links back to the requesting work. Provisioning owns the empty-server check, conflict handling, restart capacity, remote command, and health verification. The callers retain file deployment, beta evidence, and maintenance outcome verification. Jobs remain visible while waiting for players, other work, or integration configuration.

Maintenance also submits recovery operations and links each attempt to its incident. A failed Maintenance restart can hand its reserved slot to a rebuild after five minutes; unrelated recovery slots remain held. Drift campaigns follow their linked rebuild without requesting another restart. Their file evidence and history remain available after the instance is removed.

Campaign changes queued before submission can share one restart. Changes arriving after submission require a new operation. A failed or interrupted request keeps its original ID and never automatically replays. An explicit beta retry starts a new attempt, preserving prior history; paused or closed beta work cannot start a queued restart. Recovery of an already submitted restart continues even if its requester closes. Upgrades adopt existing restart intent for observation only.

Operation details

Each job page puts its recorded outcome, blocking condition, and available next action before launch files. Execution is grouped into phases, with separate milestones for remote deletion, local cleanup, and replacement creation during a rebuild. Copy the operation ID or link when investigating or handing off a job. Related work links back to the requesting campaign, beta, maintenance item, or collection. A completed provisioning job does not imply that its requester has finished its own verification.

New operations record structured verification evidence alongside their activity: the check result, source, evaluation time, and expected/observed map or Plutonium revision when relevant. Create and Rebuild show Verify Plutonium revision, with separate evidence for the production manifest and the running revision from RCON. Game status can be verified by RCON or accepted fresh telemetry. Engine-map checks do not independently verify a T6 start location. Passed checks describe their recorded observation; later checks can fail or become unknown. Completion describes what was verified at that time, while View server opens the current instance. Older history retains its recorded stages and outcomes; unavailable evidence is not inferred from stage wording.

Submission attempted, command accepted, and verified completion are separate milestones. A failed restart may still have ongoing recovery observation and hold restart capacity. Its original failure remains in history even if recovery later succeeds and releases the slot. Capacity and request order are snapshots, not guarantees of the next job admitted or an estimated completion time. Blocked requests can be bypassed by eligible work.

Recovery controls use the server's current eligibility checks. Resume health verification observes the existing instance without submitting another create or restart. Resume cleanup retries local cleanup after confirmed remote deletion, including eligible historical deleted identities; replacement or newer-operation guards still apply. Resume rebuild cleanup completes cleanup before creating the replacement in its reserved slot. Resuming the same operation preserves its earlier failure and resume events. Uncertain commands have no generic Retry action.

Create and rebuild capture a setup when queued and select the latest promoted files again when creation begins. The page distinguishes Setup captured when queued from Files selected for this attempt. A queued capture does not pin versions; an attempted selection is not proof that every file was installed. Launch another server uses the latest promoted setup with a fresh identity and port, rather than repairing or replaying the recorded job.

Five recent events remain visible. Expand full history to search, change order, and load older pages. A fixed history watermark keeps the viewed pages stable as events arrive; Jump to latest incorporates new activity. The detail view fetches a bounded recent-event window, while full history loads on demand.

The page refreshes every five seconds while visible, including terminal jobs whose recovery or available actions may still change. Page updated describes the last successful detail fetch; it is separate from the worker's evaluation and telemetry receipt times. A refresh failure keeps the last snapshot visible and does not erase an acknowledged action result. These timestamps are not worker heartbeat or completion-time promises.

Routes and compatibility

• /provisioning: active operations and history filtered by action.
• /provisioning/:id: operation stages, errors, timestamps, and activity.
• /collections: saved lineups, active launches, and launch history.
• /collections/launches/:id: collection progress, with links to each operation.
• GET /api/provisioning: lifecycle history (deployments response property).
• POST /api/provisioning: queue { "server": "t5_zombie_theater_1", "action": "start" }. Supported actions are start, restart, rebuild, and stop. Returns HTTP 202 with deploymentId. Create continues through POST /api/servers; Destroy uses DELETE /api/servers/:name.
• /api/provisioning/:id, /:id/resume, /:id/cancel, /preview, and /house-party preserve the existing operation, recovery, removal cancellation, launch preview, and fleet destruction APIs.
• GET /api/provisioning/:id?events=recent returns the same detail with five recent events plus activity totals/latest ID and a failure excerpt. Omitting the option preserves the full legacy event response. Progress, capacity, named related work, and normalized warning messages are additive fields; the original warning string and lifecycle fields remain available.
• GET /api/provisioning/:id/events reads history with order=newest|oldest, literal-text query, limit (default 25, maximum 100), an exclusive event-ID cursor, and an upper-ID watermark. Keep the returned watermark when loading more and use nextCursor until it is null. Begin without a watermark to include newly arrived events. These reads do not probe Komodo or RCON.

Old /deployments pages redirect to their /provisioning equivalents, preserving query parameters. Collection launch links at /provisioning/collections/:id or /deployments/collections/:id redirect to /collections/launches/:id and preserve query parameters. /api/deployments remains compatible, including old campaign detail endpoints. No history or IDs are renamed or discarded.

Start and Stop use Komodo's corresponding runtime commands. Rebuild uses full destruction and creation, not Komodo's redeploy command. See the Komodo execute API for the underlying container operations.