electric-cherry/docs/guides/collections.md

# Deployment collections

Open **Fleet → Collections** to manage reusable server lineups. Use
**Active launches** to follow progress and **Launch history** to review completed
launches and failures. Both views refresh automatically. A
collection describes a whole lineup: for example, two Kino lobbies, one Moon lobby,
and two Town lobbies. Counts are desired fleet totals: existing and queued
servers count toward each target, and only missing servers are created.
Collections save game/map selections and lobby counts. Every new server uses the
latest promoted scripts, plugins, baseline, and matching configuration overlays
when provisioning begins.

## How lineup counts work

For example, a collection targeting four Kino and two Der Riese lobbies, with
two Kino already registered, creates two Kino and two Der Riese lobbies.
The preview shows **Target / Existing or queued / To create** for each map.
Matching is by game and exact map variant; T6 Town and Farm are distinct even
though they share an engine map. Provisioning identities are counted once when
they appear in both the server registry and active creation queue.

Registered servers count even if stopped or unhealthy; conforming does not
restart or reconfigure them. Failed creation jobs without a registered server,
destroyed records, and servers awaiting removal do not count. The default
**Fill missing servers** mode keeps excess servers and maps outside the collection.
It fills deficits without changing existing servers' files.

## Build and launch a collection

1. Choose **Create collection** on **Fleet → Collections** and name the lineup.
2. Add maps from Black Ops I or II and set the desired total lobbies for each.
3. Choose **Save & preview** (or **Preview fleet changes** on a saved collection).
   Choose **Fill missing servers** or **Match collection exactly**, then review
   the server IDs and estimated capacity. Native host ports are assigned later.
4. Launches covered by unused keys show **Game key available**. Each game’s
   available keys are counted once across the full lineup. If more keys are
   needed, add them under **Administration → Keys**, then refresh the preview.
   Launch is enabled when every new server has an available game key.
5. Choose **Queue N servers** (or confirm removals and choose **Apply** in exact
   mode) and follow the collection progress page.
   If every target is already met, choose **Done**; no operation is created.

### Match collection exactly

Exact mode applies to the entire registered and queued fleet across both games.
The preview lists servers to keep, create, and remove. It keeps deployed lobbies
before unstarted creations, then the lowest lobby numbers within each game/map
variant. It removes excess lobbies and maps outside the collection. Retained
servers keep their containers, files, ports, and configuration without a restart.
The option is chosen per launch; opening another preview defaults to fill mode.

Review the removal list and check its confirmation before applying. Removed
servers lose their containers, files, and registrations; their keys return to the
game pool only after confirmed removal.
Unstarted creations are cancelled without booting a container. Running servers
use the normal removal queue and require a fresh zero-player RCON response.
Occupied or unreachable servers wait. Already pending removals are reused.

Creation intents and removals are queued in one database transaction. A
missing key or changed fleet rejects the entire request. Any fleet identity,
map, port, or pending-removal change since an exact preview requires a new review.
Servers added after submission are not included. Applying the same preview again
returns the same collection without repeating operations.

Keys assigned to pending removals stay reserved until cleanup is confirmed. An
exact collection that replaces running servers still needs enough currently
available game keys for every new launch. If the pool is full, complete the
removals first, then refresh the preview to use the released keys.

Native launches can be queued even when the current port range is full. Removed
servers retain their ports until cleanup completes; provisioning assigns newly
available ports as each creation starts. A failed or cancelled removal can leave
creations waiting for capacity, while other eligible hosts can still be used.
Legacy launches retain their reviewed port dependencies. Queued work and any
assigned reservations survive app restarts. The progress page
tracks creations and removals, including removal-only plans, and lists the servers
that were retained. The existing operation detail page offers removal cancellation
and cleanup recovery where applicable.

### Collection limits and editing

Collections support up to 40 distinct game/map selections, 1–20 lobbies per map,
and 100 servers total. The host's configured port range and existing fleet may
allow fewer. A collection can be saved even when there is insufficient capacity
to launch it. Saving a collection does not capture or pin managed file versions.

Use **Edit** on a collection card to change its name, description, maps, or lobby
counts. Already reviewed launches keep their original lineup, including after
collection edits or removal; their managed files still follow the latest promotions.
Previously saved collections automatically use this policy without being resaved.
Collections saved by the earlier single-server release load as one-server lineups
and can be expanded directly. Any file snapshots in older collections are ignored
when preparing new launches.

### Preview and submission

A native collection preview reads a shared capacity snapshot, maintained from host
heartbeats and fleet changes. It estimates available ports, CPU, memory, and startup
slots without running placement for each server. Missing or insufficient capacity
is a warning: provisioning can wait for it. The preview does not reserve resources.

Apply validates the reviewed configuration, lobby IDs, keys, and fleet changes, saves all jobs in
one database transaction, and returns the progress page ID. Native jobs initially
have no host or port; each job performs its capacity checks asynchronously during
provisioning. If a server lacks an available game key or its reviewed identity changed,
no jobs or key reservations from that submission are committed. Legacy submissions
also revalidate their reviewed ports. Refreshing the preview rechecks pool
availability and resets removal confirmation.

Submitting the same preview again returns the same collection, including after
a lost response or an application restart. Reusing the collection with a new
preview recomputes the shortfall, including servers queued by earlier launches.
It does not add another full copy of the lineup. If the shortfall or reviewed
identities change before submission, the request is rejected without changes and
requires a fresh preview. Previously saved additive previews are checked against
the current shortfall too.

### Startup queue and progress

All collections and individual launches share one startup queue, ordered by request
time. By default, one server starts at a time. Its startup slot remains occupied
until the host agent confirms container health, game status and the selected map
are verified, and RCON reports a current Plutonium revision.
Only then can the next queued server start. Set **Concurrent server startups** in **Administration → Settings** to
a positive integer to allow more simultaneous startups. Each worker pass admits
at most the available slots, even if an attempt fails immediately.

Names, any assigned ports, and queue order survive application restarts. A server still booting
or waiting for health keeps its slot until it passes verification or is removed.
A lost startup response also keeps the slot because the container may already be
running. A confirmed creation failure is recorded for review, allowing later
queued servers to proceed on subsequent passes without undoing successful launches.

The collection launch page at `/collections/launches/:id` separates ready,
launching, queued, and failed servers. Its breadcrumb returns to Collections. It
highlights the current startup, shows each waiting server's queue position, and
links to earlier launches when another collection occupies the available capacity.
Every server links to its Provisioning operation timeline and recovery options,
and each child operation links back to its collection launch. Individual operations
remain visible in Provisioning activity and history.

## Create a single server

1. Open **Create server** from the fleet or **Single server** from Provisioning.
2. Choose a game and map. The dialog automatically checks the current promoted
   setup, latest game files, available lobby, and game key pool as selections change.
3. Review the readiness summary and resolve any displayed blockers. When no game
   key is available, open **Administration → Keys** and add one or wait for a key
   to be released. Returning to the launch dialog rechecks availability; **Retry checks**
   also runs the checks again.
4. If needed, expand **Advanced settings** for runtime and network overrides.
   Expand **Setup details** to inspect the image, configuration, managed files,
   game-file upload time, and internal server ID.
5. Choose **Launch server** in the persistent footer and follow the operation to
   verified health. **Cancel** closes the dialog without creating a server.

The single-server dialog uses the current promoted setup. Create and launch
collections separately from **Fleet → Collections**.

Automatic checks reserve neither capacity nor a key. The reviewed preview lasts
30 minutes. If another launch takes the lobby or key, submission stops; retry the
checks before launching. Pool availability is checked again at submission, and
keys are reserved atomically so concurrent launches cannot share a credential.

Host selection, game-file versions, runtime configuration and lobby availability
are checked before queueing. Provisioning assigns host capacity and physical
ports when dispatching. The host agent performs container and RCON verification
locally after launch. See [native hosts](native-hosts.md) for readiness checks.
T6 variants retain their exact catalog slug and launch ruleset; RCON health checks
verify the engine map, not the variant's start location.

## Save and reuse a collection

Choose **Create collection** under **Fleet → Collections**, give it a name and
optional description, and add maps with their desired lobby counts. A collection
can contain a single map with a target of one lobby. You can save it even when
host configuration or available capacity blocks launching. Names are unique
without regard to case.

Each map in a collection retains the game, exact map variant, and desired lobby
count. Image references, managed files, and composed configuration are resolved
from the current setup for each new launch. Credentials, lobby IDs, and ports are
not copied from the source server.

Use **Conform fleet** on a collection card to review missing servers. Remove a
collection from the library when it is no longer needed; its existing servers,
reviewed previews, and deployment history remain intact.

On a creation detail page, **Launch another server** opens the single-server
dialog with the original game and map selected. You can change either selection
before launching. Checks use the current promoted setup and latest game files,
and provisioning assigns a fresh lobby and port. The recorded files remain
visible as deployment history. Older deployments without a recorded recipe
retain their history but cannot offer this action.

## Promotions and deployment history

A preview shows the currently promoted managed files, configuration, and image
reference for the selected map. It does not pin those versions. New launches
resolve the promoted setup again when provisioning begins, so
promotions made after a preview or while a job is queued apply to that server.
This applies to collections, single-server launches, and **Launch another server**.

Previews and queued jobs store file metadata without copying script or plugin
contents for every lobby. Provisioning captures the bytes when each job starts.
Native host ranking, file reads, and command encoding run in a worker thread.
The controller makes a short atomic reservation check after ranking, then
rechecks capacity, promotions, credentials, and safety
before submitting those commands. Agent delivery streams the stored payloads in
small chunks, while progress polling reads metadata without loading file bytes.

The files selected at provisioning, their hashes and version metadata, composed
configuration, selected map, and image reference are recorded in SQLite for
deployment history. These records do not override promotions on future launches.
Existing API clients that omit a preview and jobs queued before this update
also use the promoted setup at provisioning time.

A submitted preview is bound to one deployment: retrying the same submission
returns the original operation, including after an application restart or a lost
HTTP response. The reviewed game/map, lobby ID, and key checks are preserved;
native ports are reserved only during provisioning.
Already running servers continue to receive updates through campaigns, betas,
and drift reconciliation using the server's game/map scopes.
Container tags can resolve to newer images; use a digest for a fixed image.
Without a promoted baseline, Plutainer seeds its own defaults on first boot,
which are not captured. Host, network, game-file volumes, destination directories,
and RCON/Plutonium credentials use the environment and assigned game-pool key at
launch time.
The current main config filename and launch settings apply to new deployments.

Recipes do not include the separately stored Plutonium or integration credentials.
Managed files can themselves contain sensitive configuration, so protect database
backups as you would the existing artifact library and saved-key database. Public
recipe responses contain file metadata and hashes, not file contents.

## Progress and interrupted operations

Creation verifies container health, then RCON, then the selected engine map.
The detail page explains the current condition, displays elapsed time, and keeps
its activity history. A completed container start is not reported as a completed
server launch until the health checks pass.

For failed or interrupted creations, **Resume health verification** appears only
when the original server is still registered on its assigned port and no later
lifecycle operation has targeted that ID. It resumes observation only; it never
recreates or restarts a container. Missing servers and later destruction attempts
require inspection rather than replaying provisioning automatically.
After an application restart, the worker automatically resumes observation of
eligible interrupted startups so their occupied slots are retained. It never
repeats container creation or deployment as part of this recovery.

## API

Collections retain the existing `/api/blueprints` endpoints and `blueprintId` /
`blueprintName` fields for compatibility. Saved data needs no migration. Old
`/provisioning?view=blueprints` links redirect to `/collections`.
Collection launch details live at `/collections/launches/:id`; old
`/provisioning/collections/:id` and `/deployments/collections/:id` links redirect
there while preserving query parameters.

- `GET /api/blueprints`: list lineup summaries with game/map selections and counts.
- `POST /api/blueprints`: save `{ name, description?, previewId }` or
  `{ name, description?, deploymentId }` for a single-map target of one lobby, or
  `{ name, description?, entries: [{ game, mapSlug, count }] }` for a collection.
- `PUT /api/blueprints/:id`: edit `{ name, description?, entries }`. All entries
  use the latest promoted setup when launched.
- `DELETE /api/blueprints/:id`: remove a library entry.
- `POST /api/deployment-collections/preview`: preview `{ blueprintId, mode? }`,
  where mode is `fill` (default) or `exact`. Returns `mode`, `retainedServers`,
  `removals` (name/reason), creation `servers`, and configured `launchConcurrency`.
- `POST /api/deployment-collections`: atomically queue `{ previewId, keys? }`,
  where `keys` maps preview server IDs to optional unused keys added to their
  game's pool and assigned during launch. Omit entries covered by available
  pool keys; these IDs do not permanently bind a key to a map or lobby.
  A satisfied preview returns `{ ok: true, collectionId: null }` with HTTP 200,
  without creating a collection, saving keys, or kicking the worker.
- `GET /api/deployment-collections`: list collection progress summaries with
  separate `queued`, `provisioning`, and `verifying` counts.
- `GET /api/deployment-collections/:id`: collection mode, retained server IDs,
  per-operation progress with `action: create | destroy`,
  each waiting server's global `queuePosition`, and the shared queue's concurrency,
  active startups, and queued count.
- `POST /api/deployments/preview`: preview `{ game, mapSlug }` or `{ deploymentId }`
  for a new single server using the latest promoted setup. All collections, including one-server
  targets, must use the collection endpoint so existing servers count correctly.
  Returns an ID, expiry, recipe summary, capacity, key
  availability (never its value), and prerequisite checks.
- `POST /api/servers`: existing request plus optional `previewId`; the game and
  map must match that preview. `expectedName` remains supported.
- `GET /api/deployments/:id`: includes `recipe`, `blueprintName`, and
  `canResumeVerification` for lifecycle operations.
- `POST /api/deployments/:id/resume`: resume eligible creation health checks.

Collection launches begin processing immediately after all requests are queued.