electric-cherry/docs/guides/collections.md

# Deployment collections

Open **Fleet → Collections** to manage reusable server lineups. Use
**Active launches** to follow progress and **Launch history** to review completed
launches, cancellations, and failures. Both views refresh automatically. A
collection describes a whole lineup: for example, two Kino lobbies, one Moon lobby,
and two Town lobbies. Counts are desired fleet totals: existing and queued
servers count toward each target, and only missing servers are created.
Collections save game/map selections and lobby counts. Every new server uses the
latest promoted scripts, plugins, baseline, and matching configuration overlays
when provisioning begins.

## How lineup counts work

For example, a collection targeting four Kino and two Der Riese lobbies, with
two Kino already registered, creates two Kino and two Der Riese lobbies.
The preview groups maps by game and shows **Target / Existing or queued /
Create** for each map, with **Remove** in exact mode. Existing totals in exact
mode include the listed removals, including those already awaiting cleanup.
Matching is by game and exact map variant; T6 Town and Farm are distinct even
though they share an engine map. Provisioning identities are counted once when
they appear in both the server registry and active creation queue.

Registered servers count even if stopped or unhealthy; conforming does not
restart or reconfigure them. Failed creation jobs without a registered server,
destroyed records, and servers awaiting removal do not count. The default
**Fill missing servers** mode keeps excess servers and maps outside the collection.
It fills deficits without changing existing servers' files.

## Build and launch a collection

1. Choose **Create collection** on **Fleet → Collections** and name the lineup.
2. Add maps from Black Ops I or II and set the desired total lobbies for each.
3. Choose **Save & preview** (or **Preview fleet changes** on a saved collection).
   The dialog prepares its preview automatically. Both **Fill missing**
   and **Match exactly** are visible; selecting a mode rechecks the
   plan. Review the change summary and the map table grouped by game.
4. Check the required key coverage for each game. Available keys count once
   across the full lineup, and every new server needs a key. If a game is short,
   open **Admin → Keys** to add keys or wait for assigned keys to be
   released. Returning to the dialog rechecks the pools and fleet.
5. Review the capacity summary. Expand its details for CPU, memory, ports, and
   startup slots. Capacity estimates are advisory: queued launches can wait for
   hosts or resources. Native host ports are assigned during provisioning.
6. Expand **Server details** to inspect server IDs, launch order, retained servers,
   and existing-server health. **Advanced settings** holds the runtime override;
   **Setup details** shows configuration, managed files, and game-file upload
   information.
7. Use **Queue N servers** in the persistent footer, or review and confirm
   the removal list before choosing **Create N · Remove M** when removals are
   needed. Follow the collection progress page after submission. If no changes are needed,
   choose **Done**; no operation is created.

### Match collection exactly

Exact mode applies to the entire registered and queued fleet across both games.
The preview lists servers to keep, create, and remove. It keeps deployed lobbies
before unstarted creations, then the lowest lobby numbers within each game/map
variant. It removes excess lobbies and maps outside the collection. Retained
servers keep their containers, files, ports, and configuration without a restart.
The option is chosen per launch; opening another preview defaults to fill mode.

The removal list stays visible in exact mode. Review it and check its confirmation
before applying. Every refreshed preview clears that confirmation so it must be
reviewed again. Removed servers lose their containers, files, and registrations;
their keys return to the game pool only after confirmed removal.
Unstarted creations are cancelled without booting a container. Running servers
use the normal removal queue and require a fresh zero-player RCON response.
Occupied or unreachable servers wait. Already pending removals are reused.

Creation intents and removals are queued in one database transaction. A
missing key or changed fleet rejects the entire request. Any fleet identity,
map, port, or pending-removal change since an exact preview requires a new review.
Servers added after submission are not included. Applying the same preview again
returns the same collection without repeating operations.

Keys assigned to pending removals stay reserved until cleanup is confirmed. An
exact collection that replaces running servers still needs enough currently
available game keys for every new launch. If the pool is full, complete the
removals first, then refresh the preview to use the released keys.

Native launches can be queued even when the current port range is full. Removed
servers retain their ports until cleanup completes; provisioning assigns newly
available ports as each creation starts. A failed or cancelled removal can leave
creations waiting for capacity, while other eligible hosts can still be used.
Legacy launches retain their reviewed port dependencies. Queued work and any
assigned reservations survive app restarts. The progress page
tracks creations and removals, including removal-only plans, and lists the servers
that were retained. The existing operation detail page offers removal cancellation
and cleanup recovery where applicable.

### Collection limits and editing

Collections support up to 40 distinct game/map selections, 1–15 lobbies per map,
and 100 servers total. The host's configured port range and existing fleet may
allow fewer. A collection can be saved even when there is insufficient capacity
to launch it. Saving a collection does not capture or pin managed file versions.

Use **Edit** on a collection card to change its name, description, maps, or lobby
counts. Already reviewed launches keep their original lineup, including after
collection edits or removal; their managed files still follow the latest promotions.
Previously saved collections automatically use this policy without being resaved.
Collections saved by the earlier single-server release load as one-server lineups
and can be expanded directly. Any file snapshots in older collections are ignored
when preparing new launches.

### Preview and submission

A preview runs automatically when the dialog opens, the mode or launch settings
change, the user returns from managing keys, or the current preview expires.
Refreshing also resets removal confirmation. The preview lasts 30 minutes;
checks and displayed counts describe the fleet when it was prepared.

A native collection preview reads a shared capacity snapshot, maintained from host
heartbeats and fleet changes. It estimates available ports, CPU, memory, and startup
slots without running placement for each server. Missing or insufficient capacity
is a warning: provisioning can wait for it. The preview does not reserve resources.

Apply validates the reviewed configuration, lobby IDs, keys, and fleet changes, saves all jobs in
one database transaction, and returns the progress page ID. Native jobs initially
have no host or port; each job performs its capacity checks asynchronously during
provisioning. If a server lacks an available game key or its reviewed identity changed,
no jobs or key reservations from that submission are committed. Legacy submissions
also revalidate their reviewed ports. Refreshing the preview rechecks pool
availability and resets removal confirmation.

If a submission response is lost or cannot be confirmed, retry that same
submission. The dialog keeps the reviewed preview while the outcome is uncertain
instead of automatically replacing it. If the original request committed,
submitting the same preview again returns the same collection, including after
its original expiry or an application restart, without repeating operations.

Reusing the collection with a new preview recomputes the shortfall, including
servers queued by earlier launches. It does not add another full copy of the
lineup. If the shortfall or reviewed identities change before submission, the request is rejected without changes and
requires a fresh preview. Previously saved additive previews are checked against
the current shortfall too.

### Startup queue and progress

All collections and individual launches share one startup queue, ordered by request
time. By default, one server starts at a time. Its startup slot remains occupied
until the host agent confirms container health, game status and the selected map
are verified, and RCON reports a current Plutonium revision.
Only then can the next queued server start. Set **Concurrent server startups** in **Admin → Settings** to
a positive integer to allow more simultaneous startups. Each worker pass admits
at most the available slots, even if an attempt fails immediately.

Names, any assigned ports, and queue order survive application restarts. A server still booting
or waiting for health keeps its slot until it passes verification or is removed.
A lost startup response also keeps the slot because the container may already be
running. A confirmed creation failure is recorded for review, allowing later
queued servers to proceed on subsequent passes without undoing successful launches.

The collection launch page at `/collections/launches/:id` separates ready,
launching, queued, failed, and cancelled servers. Its breadcrumb returns to Collections. It
highlights the current startup, shows each waiting server's queue position, and
links to earlier launches when another collection occupies the available capacity.
Every server links to its Provisioning operation timeline and recovery options,
and each child operation links back to its collection launch. Individual operations
remain visible in Provisioning activity and history.

### Cancel a collection launch

Operators and administrators can choose **Cancel collection launch** on the
collection progress page while eligible work remains. Review and confirm the
cleanup before proceeding. Cancellation stops unfinished creations, including
eligible failed or interrupted launches, and removes their containers and instance
files. This can disconnect players who joined before startup verification finished,
even when player status is unknown. Shared game files are kept. Keys and ports
remain reserved until removal is confirmed.

Completed launches, including those that finished with a warning, and retained
servers are kept. In exact mode, safe pending removals belonging to this launch
are cancelled. Removals that have already started or are still needed by another
collection may finish; cancellation does not restore servers already removed.

The progress page shows **Cancellation requested** while work or cleanup remains,
then **Collection launch cancelled**, or **Cancellation needs attention** if
operations or cleanup failed. Cancelled operations are counted separately
from failures. Each cancelled creation links to its cleanup operation for progress
and recovery. A launch stays in **Active launches** until its work and cleanup
have settled; failed cleanup remains marked as needing attention in launch history.
Cancellation does not delete the saved collection, so it can be previewed again
against the remaining fleet.

## Create a single server

1. Open **Create server** from the fleet or **Single server** from Provisioning.
2. Choose a game and map. The dialog automatically checks the current promoted
   setup, latest game files, available lobby, and game key pool as selections change.
3. Review the readiness summary and resolve any displayed blockers. When no game
   key is available, open **Admin → Keys** and add one or wait for a key
   to be released. Returning to the launch dialog rechecks availability; **Retry checks**
   also runs the checks again.
4. If needed, expand **Advanced settings** for runtime and network overrides.
   Expand **Setup details** to inspect the image, configuration, managed files,
   game-file upload time, and internal server ID.
5. Choose **Launch server** in the persistent footer and follow the operation to
   verified health. **Cancel** closes the dialog without creating a server.

The single-server dialog uses the current promoted setup. Create and launch
collections separately from **Fleet → Collections**.

Automatic checks reserve neither capacity nor a key. The reviewed preview lasts
30 minutes. If another launch takes the lobby or key, submission stops; retry the
checks before launching. Pool availability is checked again at submission, and
keys are reserved atomically so concurrent launches cannot share a credential.

Host selection, game-file versions, runtime configuration and lobby availability
are checked before queueing. Provisioning assigns host capacity and physical
ports when dispatching. The host agent performs container and RCON verification
locally after launch. See [native hosts](native-hosts.md) for readiness checks.
T6 variants retain their exact catalog slug and launch ruleset; RCON health checks
verify the engine map, not the variant's start location.

## Save and reuse a collection

Choose **Create collection** under **Fleet → Collections**, give it a name and
optional description, and add maps with their desired lobby counts. A collection
can contain a single map with a target of one lobby. You can save it even when
host configuration or available capacity blocks launching. Names are unique
without regard to case.

Each map in a collection retains the game, exact map variant, and desired lobby
count. Image references, managed files, and composed configuration are resolved
from the current setup for each new launch. Credentials, lobby IDs, and ports are
not copied from the source server.

Use **Conform fleet** on a collection card to review missing servers. Remove a
collection from the library when it is no longer needed; its existing servers,
reviewed previews, and deployment history remain intact.

On a creation detail page, **Launch another server** opens the single-server
dialog with the original game and map selected. You can change either selection
before launching. Checks use the current promoted setup and latest game files,
and provisioning assigns a fresh lobby and port. The recorded files remain
visible as deployment history. Older deployments without a recorded recipe
retain their history but cannot offer this action.

## Promotions and deployment history

A preview shows the currently promoted managed files, configuration, and image
reference for the selected map. It does not pin those versions. New launches
resolve the promoted setup again when provisioning begins, so
promotions made after a preview or while a job is queued apply to that server.
This applies to collections, single-server launches, and **Launch another server**.

Previews and queued jobs store file metadata without copying script or plugin
contents for every lobby. Provisioning captures the bytes when each job starts.
Native host ranking, file reads, and command encoding run in a worker thread.
The controller makes a short atomic reservation check after ranking, then
rechecks capacity, promotions, credentials, and safety
before submitting those commands. Agent delivery streams the stored payloads in
small chunks, while progress polling reads metadata without loading file bytes.

The files selected at provisioning, their hashes and version metadata, composed
configuration, selected map, and image reference are recorded in SQLite for
deployment history. These records do not override promotions on future launches.
Existing API clients that omit a preview and jobs queued before this update
also use the promoted setup at provisioning time.

A submitted preview is bound to one deployment: retrying the same submission
returns the original operation, including after an application restart or a lost
HTTP response. The reviewed game/map, lobby ID, and key checks are preserved;
native ports are reserved only during provisioning.
Already running servers continue to receive updates through campaigns, betas,
and drift reconciliation using the server's game/map scopes.
Container tags can resolve to newer images; use a digest for a fixed image.
Without a promoted baseline, Plutainer seeds its own defaults on first boot,
which are not captured. Host, network, game-file volumes, destination directories,
and RCON/Plutonium credentials use the environment and assigned game-pool key at
launch time.
The current main config filename and launch settings apply to new deployments.

Recipes do not include the separately stored Plutonium or integration credentials.
Managed files can themselves contain sensitive configuration, so protect database
backups as you would the existing artifact library and saved-key database. Public
recipe responses contain file metadata and hashes, not file contents.

## Progress and interrupted operations

Creation verifies container health, then RCON, then the selected engine map.
The detail page explains the current condition, displays elapsed time, and keeps
its activity history. A completed container start is not reported as a completed
server launch until the health checks pass.

For failed or interrupted creations, **Resume health verification** appears only
when the original server is still registered on its assigned port and no later
lifecycle operation has targeted that ID. Native commands must have completed
before readiness verification can resume. If the container is unhealthy, the
operation may continue its remaining automatic restart and rebuild recovery steps.

When the host conclusively reports that the initial creation command failed,
**Retry with rebuild** starts a new provisioning operation for the same lobby and
port. It deletes the container and instance files and uses the latest promoted
setup. Normal player safety, capacity, and protected work checks still apply.
The original failure stays in history and links to the recovery operation.
This retry is unavailable for unknown command results, replacement instances,
cancelled creation cleanup, or a later operation on the same server.

After an application restart, the worker reconciles the original host command
before continuing eligible interrupted startups. Unknown command results retain
their original command ID and occupied slots until the host confirms the outcome.
Missing server registrations and later destruction attempts require inspection.

### Recovering a completed rebuild generation

A completed rebuild records the replacement generation before EC verifies health.
If fresh host inventory disagrees, verification pauses with the controller and
host generations in the operation's recovery message.

For an existing mismatch, open the **completed rebuild's Provisioning detail**
(even if that operation was later cancelled), then choose **Reconcile replacement
generation**. EC requires a completed command for the original host and immutable
instance ID, the command's requested replacement generation, and matching host
inventory observed within the last 30 seconds. Wait for that host to reconnect
and report fresh inventory if the action is unavailable. A server name alone is
not sufficient evidence.

Reconciliation preserves cancellation and submitted commands. Use **Resume health
verification** when offered, or request a new lifecycle operation after resolving
any remaining active operation. Fresh health checks, log reads, and new lifecycle
commands use the replacement generation. EC requires fresh container and game
evidence before reporting the server online. The agent continues to reject stale
commands; an admin player-safety override does not bypass generation checks.

## API

Collections retain the existing `/api/blueprints` endpoints and `blueprintId` /
`blueprintName` fields for compatibility. Saved data needs no migration. Old
`/provisioning?view=blueprints` links redirect to `/collections`.
Collection launch details live at `/collections/launches/:id`; old
`/provisioning/collections/:id` and `/deployments/collections/:id` links redirect
there while preserving query parameters.

- `GET /api/blueprints`: list lineup summaries with game/map selections and counts.
- `POST /api/blueprints`: save `{ name, description?, previewId }` or
  `{ name, description?, deploymentId }` for a single-map target of one lobby, or
  `{ name, description?, entries: [{ game, mapSlug, count }] }` for a collection.
- `PUT /api/blueprints/:id`: edit `{ name, description?, entries }`. All entries
  use the latest promoted setup when launched.
- `DELETE /api/blueprints/:id`: remove a library entry.
- `POST /api/deployment-collections/preview`: preview `{ blueprintId, mode? }`,
  where mode is `fill` (default) or `exact`. Returns `mode`, `retainedServers`,
  `removals` (name/reason), creation `servers`, and configured `launchConcurrency`.
- `POST /api/deployment-collections`: atomically queue `{ previewId, keys? }`,
  where `keys` maps preview server IDs to optional unused keys added to their
  game's pool and assigned during launch. Omit entries covered by available
  pool keys; these IDs do not permanently bind a key to a map or lobby.
  A satisfied preview returns `{ ok: true, collectionId: null }` with HTTP 200,
  without creating a collection, saving keys, or kicking the worker.
- `GET /api/deployment-collections`: list collection progress summaries with
  separate `queued`, `provisioning`, `verifying`, and `cancelled` counts,
  `cleanupActive` and `cleanupFailed` counts, and `cancelled_at` when cancellation
  was requested. Cancelled operations are excluded from `failed`.
- `GET /api/deployment-collections/:id`: collection mode, retained server IDs,
  per-operation progress with `action: create | destroy`,
  each waiting server's global `queuePosition`, and the shared queue's concurrency,
  active startups, and queued count. Includes `canCancel` and each creation's
  `cancellation` cleanup ID, state, stage, and error when applicable.
- `POST /api/deployment-collections/:id/cancel`: request cancellation and cleanup
  of eligible unfinished work. Returns `{ ok: true }`, HTTP 404 for an unknown
  launch, or HTTP 409 when the launch cannot be cancelled.
- `POST /api/deployments/preview`: preview `{ game, mapSlug }` or `{ deploymentId }`
  for a new single server using the latest promoted setup. All collections, including one-server
  targets, must use the collection endpoint so existing servers count correctly.
  Returns an ID, expiry, recipe summary, capacity, key
  availability (never its value), and prerequisite checks.
- `POST /api/servers`: existing request plus optional `previewId`; the game and
  map must match that preview. `expectedName` remains supported.
- `GET /api/deployments/:id`: includes `recipe`, `blueprintName`, and
  `canResumeVerification` and `canRetryWithRebuild` for lifecycle operations.
- `POST /api/deployments/:id/resume`: resume eligible creation health checks.
- `POST /api/provisioning/:id/reconcile-generation`: reconcile a completed rebuild
  using fresh inventory from its original host. Returns 409 when identity or
  completion evidence cannot be validated; repeated application is safe.
- `POST /api/provisioning/:id/retry-rebuild`: request a new rebuild for an eligible
  failed creation and link it from the original operation.

Collection launches begin processing immediately after all requests are queued.