electric-cherry/docs/guides/collections.md

Deployment collections

Open Fleet → Collections to manage reusable server lineups. Use Active launches to follow progress and Launch history to review completed launches, cancellations, and failures. Both views refresh automatically. A collection describes a whole lineup: for example, two Kino lobbies, one Moon lobby, and two Town lobbies. Counts are desired fleet totals: existing and queued servers count toward each target, and only missing servers are created. Collections save game/map selections and lobby counts. Every new server uses the latest promoted scripts, plugins, baseline, and matching configuration overlays when provisioning begins.

How lineup counts work

For example, a collection targeting four Kino and two Der Riese lobbies, with two Kino already registered, creates two Kino and two Der Riese lobbies. The preview groups maps by game and shows Target / Existing or queued / Create for each map, with Remove in exact mode. Existing totals in exact mode include the listed removals, including those already awaiting cleanup. Matching is by game and exact map variant; T6 Town and Farm are distinct even though they share an engine map. Provisioning identities are counted once when they appear in both the server registry and active creation queue.

Registered servers count even if stopped or unhealthy; conforming does not restart or reconfigure them. Failed creation jobs without a registered server, destroyed records, and servers awaiting removal do not count. The default Fill missing servers mode keeps excess servers and maps outside the collection. It fills deficits without changing existing servers' files.

Build and launch a collection

1. Choose Create collection on Fleet → Collections and name the lineup.
2. Add maps from Black Ops I or II and set the desired total lobbies for each.
3. Choose Save & preview (or Preview fleet changes on a saved collection). The dialog prepares its preview automatically. Both Fill missing and Match exactly are visible; selecting a mode rechecks the plan. Review the change summary and the map table grouped by game.
4. Check the required key coverage for each game. Available keys count once across the full lineup, and every new server needs a key. If a game is short, open Admin → Keys to add keys or wait for assigned keys to be released. Returning to the dialog rechecks the pools and fleet.
5. Review the capacity summary. Expand its details for CPU, memory, ports, and startup slots. Capacity estimates are advisory: queued launches can wait for hosts or resources. Native host ports are assigned during provisioning.
6. Expand Server details to inspect server IDs, launch order, retained servers, and existing-server health. Advanced settings holds the runtime override; Setup details shows configuration, managed files, and game-file upload information.
7. Use Queue N servers in the persistent footer, or review and confirm the removal list before choosing Create N · Remove M when removals are needed. Follow the collection progress page after submission. If no changes are needed, choose Done; no operation is created.

Match collection exactly

Exact mode applies to the entire registered and queued fleet across both games. The preview lists servers to keep, create, and remove. It keeps deployed lobbies before unstarted creations, then the lowest lobby numbers within each game/map variant. It removes excess lobbies and maps outside the collection. Retained servers keep their containers, files, ports, and configuration without a restart. The option is chosen per launch; opening another preview defaults to fill mode.

The removal list stays visible in exact mode. Review it and check its confirmation before applying. Every refreshed preview clears that confirmation so it must be reviewed again. Removed servers lose their containers, files, and registrations; their keys return to the game pool only after confirmed removal. Unstarted creations are cancelled without booting a container. Running servers use the normal removal queue and require a fresh zero-player RCON response. Occupied or unreachable servers wait. Already pending removals are reused.

Creation intents and removals are queued in one database transaction. A missing key or changed fleet rejects the entire request. Any fleet identity, map, port, or pending-removal change since an exact preview requires a new review. Servers added after submission are not included. Applying the same preview again returns the same collection without repeating operations.

Keys assigned to pending removals stay reserved until cleanup is confirmed. An exact collection that replaces running servers still needs enough currently available game keys for every new launch. If the pool is full, complete the removals first, then refresh the preview to use the released keys.

Native launches can be queued even when the current port range is full. Removed servers retain their ports until cleanup completes; provisioning assigns newly available ports as each creation starts. A failed or cancelled removal can leave creations waiting for capacity, while other eligible hosts can still be used. Legacy launches retain their reviewed port dependencies. Queued work and any assigned reservations survive app restarts. The progress page tracks creations and removals, including removal-only plans, and lists the servers that were retained. The existing operation detail page offers removal cancellation and cleanup recovery where applicable.

Collection limits and editing

Collections support up to 40 distinct game/map selections, 1–15 lobbies per map, and 100 servers total. The host's configured port range and existing fleet may allow fewer. A collection can be saved even when there is insufficient capacity to launch it. Saving a collection does not capture or pin managed file versions.

Use Edit on a collection card to change its name, description, maps, or lobby counts. Already reviewed launches keep their original lineup, including after collection edits or removal; their managed files still follow the latest promotions. Previously saved collections automatically use this policy without being resaved. Collections saved by the earlier single-server release load as one-server lineups and can be expanded directly. Any file snapshots in older collections are ignored when preparing new launches.

Preview and submission

A preview runs automatically when the dialog opens, the mode or launch settings change, the user returns from managing keys, or the current preview expires. Refreshing also resets removal confirmation. The preview lasts 30 minutes; checks and displayed counts describe the fleet when it was prepared.

A native collection preview reads a shared capacity snapshot, maintained from host heartbeats and fleet changes. It estimates available ports, CPU, memory, and startup slots without running placement for each server. Missing or insufficient capacity is a warning: provisioning can wait for it. The preview does not reserve resources.

Apply validates the reviewed configuration, lobby IDs, keys, and fleet changes, saves all jobs in one database transaction, and returns the progress page ID. Native jobs initially have no host or port; each job performs its capacity checks asynchronously during provisioning. If a server lacks an available game key or its reviewed identity changed, no jobs or key reservations from that submission are committed. Legacy submissions also revalidate their reviewed ports. Refreshing the preview rechecks pool availability and resets removal confirmation.

If a submission response is lost or cannot be confirmed, retry that same submission. The dialog keeps the reviewed preview while the outcome is uncertain instead of automatically replacing it. If the original request committed, submitting the same preview again returns the same collection, including after its original expiry or an application restart, without repeating operations.

Reusing the collection with a new preview recomputes the shortfall, including servers queued by earlier launches. It does not add another full copy of the lineup. If the shortfall or reviewed identities change before submission, the request is rejected without changes and requires a fresh preview. Previously saved additive previews are checked against the current shortfall too.

Startup queue and progress

All collections and individual launches share one startup queue, ordered by request time. By default, one server starts at a time. Its startup slot remains occupied until the host agent confirms container health, game status and the selected map are verified, and RCON reports a current Plutonium revision. Only then can the next queued server start. Set Concurrent server startups in Admin → Settings to a positive integer to allow more simultaneous startups. Each worker pass admits at most the available slots, even if an attempt fails immediately.

Names, any assigned ports, and queue order survive application restarts. A server still booting or waiting for health keeps its slot until it passes verification or is removed. A lost startup response also keeps the slot because the container may already be running. A confirmed creation failure is recorded for review, allowing later queued servers to proceed on subsequent passes without undoing successful launches.

The collection launch page at /collections/launches/:id separates ready, launching, queued, failed, and cancelled servers. Its breadcrumb returns to Collections. It highlights the current startup, shows each waiting server's queue position, and links to earlier launches when another collection occupies the available capacity. Every server links to its Provisioning operation timeline and recovery options, and each child operation links back to its collection launch. Individual operations remain visible in Provisioning activity and history.

Cancel a collection launch

Operators and administrators can choose Cancel collection launch on the collection progress page while eligible work remains. Review and confirm the cleanup before proceeding. Cancellation stops unfinished creations, including eligible failed or interrupted launches, and removes their containers and instance files. This can disconnect players who joined before startup verification finished, even when player status is unknown. Shared game files are kept. Keys and ports remain reserved until removal is confirmed.

Completed launches, including those that finished with a warning, and retained servers are kept. In exact mode, safe pending removals belonging to this launch are cancelled. Removals that have already started or are still needed by another collection may finish; cancellation does not restore servers already removed.

The progress page shows Cancellation requested while work or cleanup remains, then Collection launch cancelled, or Cancellation needs attention if operations or cleanup failed. Cancelled operations are counted separately from failures. Each cancelled creation links to its cleanup operation for progress and recovery. A launch stays in Active launches until its work and cleanup have settled; failed cleanup remains marked as needing attention in launch history. Cancellation does not delete the saved collection, so it can be previewed again against the remaining fleet.

Create a single server

1. Open Create server from the fleet or Single server from Provisioning.
2. Choose a game and map. The dialog automatically checks the current promoted setup, latest game files, available lobby, and game key pool as selections change.
3. Review the readiness summary and resolve any displayed blockers. When no game key is available, open Admin → Keys and add one or wait for a key to be released. Returning to the launch dialog rechecks availability; Retry checks also runs the checks again.
4. If needed, expand Advanced settings for runtime and network overrides. Expand Setup details to inspect the image, configuration, managed files, game-file upload time, and internal server ID.
5. Choose Launch server in the persistent footer and follow the operation to verified health. Cancel closes the dialog without creating a server.

The single-server dialog uses the current promoted setup. Create and launch collections separately from Fleet → Collections.

Automatic checks reserve neither capacity nor a key. The reviewed preview lasts 30 minutes. If another launch takes the lobby or key, submission stops; retry the checks before launching. Pool availability is checked again at submission, and keys are reserved atomically so concurrent launches cannot share a credential.

Host selection, game-file versions, runtime configuration and lobby availability are checked before queueing. Provisioning assigns host capacity and physical ports when dispatching. The host agent performs container and RCON verification locally after launch. See native hosts for readiness checks. T6 variants retain their exact catalog slug and launch ruleset; RCON health checks verify the engine map, not the variant's start location.

Save and reuse a collection

Choose Create collection under Fleet → Collections, give it a name and optional description, and add maps with their desired lobby counts. A collection can contain a single map with a target of one lobby. You can save it even when host configuration or available capacity blocks launching. Names are unique without regard to case.

Each map in a collection retains the game, exact map variant, and desired lobby count. Image references, managed files, and composed configuration are resolved from the current setup for each new launch. Credentials, lobby IDs, and ports are not copied from the source server.

Use Conform fleet on a collection card to review missing servers. Remove a collection from the library when it is no longer needed; its existing servers, reviewed previews, and deployment history remain intact.

On a creation detail page, Launch another server opens the single-server dialog with the original game and map selected. You can change either selection before launching. Checks use the current promoted setup and latest game files, and provisioning assigns a fresh lobby and port. The recorded files remain visible as deployment history. Older deployments without a recorded recipe retain their history but cannot offer this action.

Promotions and deployment history

A preview shows the currently promoted managed files, configuration, and image reference for the selected map. It does not pin those versions. New launches resolve the promoted setup again when provisioning begins, so promotions made after a preview or while a job is queued apply to that server. This applies to collections, single-server launches, and Launch another server.

Previews and queued jobs store file metadata without copying script or plugin contents for every lobby. Provisioning captures the bytes when each job starts. Native host ranking, file reads, and command encoding run in a worker thread. The controller makes a short atomic reservation check after ranking, then rechecks capacity, promotions, credentials, and safety before submitting those commands. Agent delivery streams the stored payloads in small chunks, while progress polling reads metadata without loading file bytes.

The files selected at provisioning, their hashes and version metadata, composed configuration, selected map, and image reference are recorded in SQLite for deployment history. These records do not override promotions on future launches. Existing API clients that omit a preview and jobs queued before this update also use the promoted setup at provisioning time.

A submitted preview is bound to one deployment: retrying the same submission returns the original operation, including after an application restart or a lost HTTP response. The reviewed game/map, lobby ID, and key checks are preserved; native ports are reserved only during provisioning. Already running servers continue to receive updates through campaigns, betas, and drift reconciliation using the server's game/map scopes. Container tags can resolve to newer images; use a digest for a fixed image. Without a promoted baseline, Plutainer seeds its own defaults on first boot, which are not captured. Host, network, game-file volumes, destination directories, and RCON/Plutonium credentials use the environment and assigned game-pool key at launch time. The current main config filename and launch settings apply to new deployments.

Recipes do not include the separately stored Plutonium or integration credentials. Managed files can themselves contain sensitive configuration, so protect database backups as you would the existing artifact library and saved-key database. Public recipe responses contain file metadata and hashes, not file contents.

Progress and interrupted operations

Creation verifies container health, then RCON, then the selected engine map. The detail page explains the current condition, displays elapsed time, and keeps its activity history. A completed container start is not reported as a completed server launch until the health checks pass.

For failed or interrupted creations, Resume health verification appears only when the original server is still registered on its assigned port and no later lifecycle operation has targeted that ID. Native commands must have completed before readiness verification can resume. If the container is unhealthy, the operation may continue its remaining automatic restart and rebuild recovery steps.

When the host conclusively reports that the initial creation command failed, Retry with rebuild starts a new provisioning operation for the same lobby and port. It deletes the container and instance files and uses the latest promoted setup. Normal player safety, capacity, and protected work checks still apply. The original failure stays in history and links to the recovery operation. This retry is unavailable for unknown command results, replacement instances, cancelled creation cleanup, or a later operation on the same server.

After an application restart, the worker reconciles the original host command before continuing eligible interrupted startups. Unknown command results retain their original command ID and occupied slots until the host confirms the outcome. Missing server registrations and later destruction attempts require inspection.

Recovering a completed rebuild generation

A completed rebuild records the replacement generation before EC verifies health. If fresh host inventory disagrees, verification pauses with the controller and host generations in the operation's recovery message.

For an existing mismatch, open the completed rebuild's Provisioning detail (even if that operation was later cancelled), then choose Reconcile replacement generation. EC requires a completed command for the original host and immutable instance ID, the command's requested replacement generation, and matching host inventory observed within the last 30 seconds. Wait for that host to reconnect and report fresh inventory if the action is unavailable. A server name alone is not sufficient evidence.

Reconciliation preserves cancellation and submitted commands. Use Resume health verification when offered, or request a new lifecycle operation after resolving any remaining active operation. Fresh health checks, log reads, and new lifecycle commands use the replacement generation. EC requires fresh container and game evidence before reporting the server online. The agent continues to reject stale commands; an admin player-safety override does not bypass generation checks.

API

Collections retain the existing /api/blueprints endpoints and blueprintId / blueprintName fields for compatibility. Saved data needs no migration. Old /provisioning?view=blueprints links redirect to /collections. Collection launch details live at /collections/launches/:id; old /provisioning/collections/:id and /deployments/collections/:id links redirect there while preserving query parameters.

• GET /api/blueprints: list lineup summaries with game/map selections and counts.
• POST /api/blueprints: save { name, description?, previewId } or { name, description?, deploymentId } for a single-map target of one lobby, or { name, description?, entries: [{ game, mapSlug, count }] } for a collection.
• PUT /api/blueprints/:id: edit { name, description?, entries }. All entries use the latest promoted setup when launched.
• DELETE /api/blueprints/:id: remove a library entry.
• POST /api/deployment-collections/preview: preview { blueprintId, mode? }, where mode is fill (default) or exact. Returns mode, retainedServers, removals (name/reason), creation servers, and configured launchConcurrency.
• POST /api/deployment-collections: atomically queue { previewId, keys? }, where keys maps preview server IDs to optional unused keys added to their game's pool and assigned during launch. Omit entries covered by available pool keys; these IDs do not permanently bind a key to a map or lobby. A satisfied preview returns { ok: true, collectionId: null } with HTTP 200, without creating a collection, saving keys, or kicking the worker.
• GET /api/deployment-collections: list collection progress summaries with separate queued, provisioning, verifying, and cancelled counts, cleanupActive and cleanupFailed counts, and cancelled_at when cancellation was requested. Cancelled operations are excluded from failed.
• GET /api/deployment-collections/:id: collection mode, retained server IDs, per-operation progress with action: create | destroy, each waiting server's global queuePosition, and the shared queue's concurrency, active startups, and queued count. Includes canCancel and each creation's cancellation cleanup ID, state, stage, and error when applicable.
• POST /api/deployment-collections/:id/cancel: request cancellation and cleanup of eligible unfinished work. Returns { ok: true }, HTTP 404 for an unknown launch, or HTTP 409 when the launch cannot be cancelled.
• POST /api/deployments/preview: preview { game, mapSlug } or { deploymentId } for a new single server using the latest promoted setup. All collections, including one-server targets, must use the collection endpoint so existing servers count correctly. Returns an ID, expiry, recipe summary, capacity, key availability (never its value), and prerequisite checks.
• POST /api/servers: existing request plus optional previewId; the game and map must match that preview. expectedName remains supported.
• GET /api/deployments/:id: includes recipe, blueprintName, and canResumeVerification and canRetryWithRebuild for lifecycle operations.
• POST /api/deployments/:id/resume: resume eligible creation health checks.
• POST /api/provisioning/:id/reconcile-generation: reconcile a completed rebuild using fresh inventory from its original host. Returns 409 when identity or completion evidence cannot be validated; repeated application is safe.
• POST /api/provisioning/:id/retry-rebuild: request a new rebuild for an eligible failed creation and link it from the original operation.

Collection launches begin processing immediately after all requests are queued.