electric-cherry/docs/deployment-blueprints.md

# Deployment collections

Open **Operations → Collections** to manage reusable server lineups. A
collection describes a whole lineup: for example, two Kino lobbies, one Moon lobby,
and two Town lobbies. Counts are desired fleet totals: existing and queued
servers count toward each target, and only missing servers are created.
Collections save game/map selections and lobby counts. Every new server uses the
latest promoted scripts, plugins, baseline, and matching configuration overlays
when provisioning begins.

For example, a collection targeting four Kino and two Der Riese lobbies, with
two Kino already registered, creates two Kino and two Der Riese lobbies.
The preview shows **Target / Existing or queued / To create** for each map.
Matching is by game and exact map variant; T6 Town and Farm are distinct even
though they share an engine map. Provisioning identities are counted once when
they appear in both the server registry and active creation queue.

Registered servers count even if stopped or unhealthy; conforming does not
restart or reconfigure them. Failed creation jobs without a registered server,
destroyed records, and servers awaiting removal do not count. The default
**Fill missing servers** mode keeps excess servers and maps outside the collection.
It fills deficits without changing existing servers' files.

## Build and launch a collection

1. Choose **Create collection** on **Operations → Collections** and name the lineup.
2. Add maps from Black Ops I or II and set the desired total lobbies for each.
3. Choose **Save & preview** (or **Preview fleet changes** on a saved collection).
   Choose **Fill missing servers** or **Match collection exactly**, then review
   the server IDs and game ports.
4. Lobbies with a saved key show **Saved key ready** and reuse it automatically.
   Enter a Plutonium key only for lobbies missing one. Update saved keys in **Keys**.
5. Choose **Queue N servers** (or confirm removals and choose **Apply** in exact
   mode) and follow the collection progress page.
   If every target is already met, choose **Done**; no operation is created.

### Match collection exactly

Exact mode applies to the entire registered and queued fleet across both games.
The preview lists servers to keep, create, and remove. It keeps deployed lobbies
before unstarted creations, then the lowest lobby numbers within each game/map
variant. It removes excess lobbies and maps outside the collection. Retained
servers keep their containers, files, ports, and configuration without a restart.
The option is chosen per launch; opening another preview defaults to fill mode.

Review the removal list and check its confirmation before applying. Removed
servers lose their containers, files, and registrations; saved keys remain.
Unstarted creations are cancelled without booting a container. Running servers
use the normal removal queue and require a fresh zero-player RCON response.
Occupied or unreachable servers wait. Already pending removals are reused.

Creation reservations and removals are queued in one database transaction. A
missing key or changed fleet rejects the entire request. Any fleet identity,
map, port, or pending-removal change since an exact preview requires a new review.
Servers added after submission are not included. Applying the same preview again
returns the same collection without repeating operations.

Ports belonging to reviewed removals can be reserved for new lobbies even when
the current port range is full. A new lobby waits until its port's removal has
completed; a failed, interrupted, or cancelled removal keeps that launch blocked,
with the reason shown in its progress. Other launches with available capacity can
proceed. Reservations and dependencies survive app restarts. The progress page
tracks creations and removals, including removal-only plans, and lists the servers
that were retained. The existing operation detail page offers removal cancellation
and cleanup recovery where applicable.

Collections support up to 40 distinct game/map selections, 1–20 lobbies per map,
and 100 servers total. The host's configured port range and existing fleet may
allow fewer. A collection can be saved even when there is insufficient capacity
to launch it. Saving a collection does not capture or pin managed file versions.

Use **Edit** on a collection card to change its name, description, maps, or lobby
counts. Already reviewed launches keep their original lineup, including after
collection edits or removal; their managed files still follow the latest promotions.
Previously saved collections automatically use this policy without being resaved.
Collections saved by the earlier single-server release load as one-server lineups
and can be expanded directly. Any file snapshots in older collections are ignored
when preparing new launches.

A collection preview does not reserve capacity. At submission, all lobby IDs,
ports, and keys are validated again. The entire collection is then queued in one
database transaction. If any server lacks a valid key or capacity has changed,
no jobs, reservations, or key changes from that submission are committed. Refreshing
the preview retains typed keys only for unchanged server IDs still missing a saved
key, and resets removal confirmation.

Submitting the same preview again returns the same collection, including after
a lost response or an application restart. Reusing the collection with a new
preview recomputes the shortfall, including servers queued by earlier launches.
It does not add another full copy of the lineup. If the shortfall or assigned
capacity changes before submission, the request is rejected without changes and
requires a fresh preview. Previously saved additive previews are checked against
the current shortfall too.

All collections and individual launches share one startup queue, ordered by request
time. By default, one server starts at a time. Its startup slot remains occupied
until Komodo is healthy, RCON answers, and the reported map matches the selection.
Only then can the next queued server start. Set **Concurrent server startups** in **Settings → Fleet settings** to
a positive integer to allow more simultaneous startups. Each worker pass admits
at most the available slots, even if an attempt fails immediately.

Names, ports, and queue order survive application restarts. A server still booting
or waiting for health keeps its slot until it passes verification or is removed.
A lost startup response also keeps the slot because the container may already be
running. A confirmed creation failure is recorded for review, allowing later
queued servers to proceed on subsequent passes without undoing successful launches.

The collection page separates ready, launching, queued, and failed servers. It
highlights the current startup, shows each waiting server's queue position, and
links to earlier launches when another collection occupies the available capacity.
Every server links to its operation timeline and recovery options, and each child
operation links back to its collection.

## Create and review a launch

1. Choose **Single server** on Provisioning (or the deploy action on the fleet).
2. Choose the current promoted setup and select a game/map, or select a collection.
3. Choose **Preview launch**. Review the image reference, configuration sources,
   managed files and versions, selected map, lobby, port, and prerequisite checks.
4. Enter the Plutonium key for the displayed server ID, or reuse its saved key.
5. Choose **Launch server** and follow the operation to verified health.

A preview lasts 30 minutes. It neither reserves capacity nor saves a key. If
another launch takes the lobby or port, submission stops before saving the key;
refresh the preview and review the newly assigned ID. Refreshing a preview clears
the typed key so it cannot accidentally be saved against a different lobby.

Komodo connection settings, target host, RCON configuration, config filename,
lobby availability, and port availability are checked locally. These checks do
not probe the host or verify game/DLC files, images, Docker volumes, firewall
rules, or credentials. Container and RCON verification happen after provisioning.
T6 variants retain their exact catalog slug and launch ruleset; RCON health checks
verify the engine map, not the variant's start location.

## Save and reuse a collection

Expand **Make this a reusable collection** in a launch preview and give it a name
and optional description. You can save the map even when host configuration or
available capacity blocks launching. Names are unique without regard to case.

Each map in a collection retains the game, exact map variant, and desired lobby
count. Image references, managed files, and composed configuration are resolved
from the current setup for each new launch. Credentials, lobby IDs, and ports are
not copied from the source server.

Use **Conform fleet** on a collection card to review missing servers. Remove a
collection from the library when it is no longer needed; its existing servers,
reviewed previews, and deployment history remain intact.

On a creation detail page, **Launch again** selects the original game and map
with the next available lobby and port, using the latest promoted setup. The
preview also lets you save that map as a collection with a target of one lobby.
The recorded files remain visible as deployment history. Older deployments
without a recorded recipe retain their history but cannot offer this action.

## Promotions and deployment history

A preview shows the currently promoted managed files, configuration, and image
reference for the selected map. It does not pin those versions. New launches
resolve the promoted setup again when submitted and when provisioning begins, so
promotions made after a preview or while a job is queued apply to that server.
This applies to collection collections, single-server launches, and **Launch again**.

The files selected at provisioning, their hashes and version metadata, composed
configuration, selected map, and image reference are recorded in SQLite for
deployment history. These records do not override promotions on future launches.
Existing API clients that omit a preview and jobs queued before this update
also use the promoted setup at provisioning time.

A submitted preview is bound to one deployment: retrying the same submission
returns the original operation, including after an application restart or a lost
HTTP response. The reviewed game/map, lobby ID, port, and key checks are preserved.
Already running servers continue to receive updates through campaigns, betas,
and drift reconciliation using the server's game/map scopes.
Container tags can resolve to newer images; use a digest for a fixed image.
Without a promoted baseline, Plutainer seeds its own defaults on first boot,
which are not captured. Host, network, game-file volumes, destination directories,
and RCON/Plutonium credentials use the environment and key store at launch time.
The current main config filename and launch settings apply to new deployments.

Recipes do not include the separately stored Plutonium or integration credentials.
Managed files can themselves contain sensitive configuration, so protect database
backups as you would the existing artifact library and saved-key database. Public
recipe responses contain file metadata and hashes, not file contents.

## Progress and interrupted operations

Creation verifies container health, then RCON, then the selected engine map.
The detail page explains the current condition, displays elapsed time, and keeps
its activity history. A completed container start is not reported as a completed
server launch until the health checks pass.

For failed or interrupted creations, **Resume health verification** appears only
when the original server is still registered on its assigned port and no later
lifecycle operation has targeted that ID. It resumes observation only; it never
recreates or restarts a container. Missing servers and later destruction attempts
require inspection rather than replaying provisioning automatically.
After an application restart, the worker automatically resumes observation of
eligible interrupted startups so their occupied slots are retained. It never
repeats container creation or deployment as part of this recovery.

## API

Collections retain the existing `/api/blueprints` endpoints and `blueprintId` /
`blueprintName` fields for compatibility. Saved data needs no migration. Old
`/provisioning?view=blueprints` links redirect to `/collections`.

- `GET /api/blueprints`: list lineup summaries with game/map selections and counts.
- `POST /api/blueprints`: save `{ name, description?, previewId }` or
  `{ name, description?, deploymentId }` for a single-map target of one lobby, or
  `{ name, description?, entries: [{ game, mapSlug, count }] }` for a collection.
- `PUT /api/blueprints/:id`: edit `{ name, description?, entries }`. All entries
  use the latest promoted setup when launched.
- `DELETE /api/blueprints/:id`: remove a library entry.
- `POST /api/deployment-collections/preview`: preview `{ blueprintId, mode? }`,
  where mode is `fill` (default) or `exact`. Returns `mode`, `retainedServers`,
  `removals` (name/reason), creation `servers`, and configured `launchConcurrency`.
- `POST /api/deployment-collections`: atomically queue `{ previewId, keys? }`,
  where `keys` maps exact preview server IDs to optional replacement keys.
  A satisfied preview returns `{ ok: true, collectionId: null }` with HTTP 200,
  without creating a collection, saving keys, or kicking the worker.
- `GET /api/deployment-collections`: list collection progress summaries with
  separate `queued`, `provisioning`, and `verifying` counts.
- `GET /api/deployment-collections/:id`: collection mode, retained server IDs,
  per-operation progress with `action: create | destroy`,
  each waiting server's global `queuePosition`, and the shared queue's concurrency,
  active startups, and queued count.
- `POST /api/deployments/preview`: preview `{ game, mapSlug }` or `{ deploymentId }`
  for a new single server using the latest promoted setup. All collections, including one-server
  targets, must use the collection endpoint so existing servers count correctly.
  Returns an ID, expiry, recipe summary, capacity, key
  availability (never its value), and prerequisite checks.
- `POST /api/servers`: existing request plus optional `previewId`; the game and
  map must match that preview. `expectedName` remains supported.
- `GET /api/deployments/:id`: includes `recipe`, `blueprintName`, and
  `canResumeVerification` for lifecycle operations.
- `POST /api/deployments/:id/resume`: resume eligible creation health checks.

Scheduled game nights and beta-gated expansion remain future layers. Collection
launches begin processing immediately after all requests are queued.