Queue native collection launches before async placement

docs/deployment-blueprints.md

@@ -29,7 +29,7 @@ It fills deficits without changing existing servers' files.
 2. Add maps from Black Ops I or II and set the desired total lobbies for each.
 3. Choose **Save & preview** (or **Preview fleet changes** on a saved collection).
    Choose **Fill missing servers** or **Match collection exactly**, then review
-   the server IDs and game ports.
+   the server IDs and estimated capacity. Native host ports are assigned later.
 4. Lobbies with a saved key show **Saved key ready** and reuse it automatically.
    Enter a Plutonium key only for lobbies missing one. Update saved keys in **Keys**.
 5. Choose **Queue N servers** (or confirm removals and choose **Apply** in exact
@@ -51,17 +51,18 @@ Unstarted creations are cancelled without booting a container. Running servers
 use the normal removal queue and require a fresh zero-player RCON response.
 Occupied or unreachable servers wait. Already pending removals are reused.
 
-Creation reservations and removals are queued in one database transaction. A
+Creation intents and removals are queued in one database transaction. A
 missing key or changed fleet rejects the entire request. Any fleet identity,
 map, port, or pending-removal change since an exact preview requires a new review.
 Servers added after submission are not included. Applying the same preview again
 returns the same collection without repeating operations.
 
-Ports belonging to reviewed removals can be reserved for new lobbies even when
-the current port range is full. A new lobby waits until its port's removal has
-completed; a failed, interrupted, or cancelled removal keeps that launch blocked,
-with the reason shown in its progress. Other launches with available capacity can
-proceed. Reservations and dependencies survive app restarts. The progress page
+Native launches can be queued even when the current port range is full. Removed
+servers retain their ports until cleanup completes; provisioning assigns newly
+available ports as each creation starts. A failed or cancelled removal can leave
+creations waiting for capacity, while other eligible hosts can still be used.
+Legacy launches retain their reviewed port dependencies. Queued work and any
+assigned reservations survive app restarts. The progress page
 tracks creations and removals, including removal-only plans, and lists the servers
 that were retained. The existing operation detail page offers removal cancellation
 and cleanup recovery where applicable.
@@ -79,18 +80,25 @@ Collections saved by the earlier single-server release load as one-server lineup
 and can be expanded directly. Any file snapshots in older collections are ignored
 when preparing new launches.
 
-A collection preview does not reserve capacity. At submission, all lobby IDs,
-ports, and keys are validated again. The entire collection is then queued in one
-database transaction. If any server lacks a valid key or capacity has changed,
-no jobs, reservations, or key changes from that submission are committed. Refreshing
+A native collection preview reads a shared capacity snapshot, maintained from host
+heartbeats and fleet changes. It estimates available ports, CPU, memory, and startup
+slots without running placement for each server. Missing or insufficient capacity
+is a warning: provisioning can wait for it. The preview does not reserve resources.
+
+Apply validates the reviewed configuration, lobby IDs, keys, and fleet changes, saves all jobs in
+one database transaction, and returns the progress page ID. Native jobs initially
+have no host or port; each job performs its capacity checks asynchronously during
+provisioning. If a server lacks a valid key or its reviewed identity changed,
+no jobs or key changes from that submission are committed. Legacy submissions also
+revalidate their reviewed ports. Refreshing
 the preview retains typed keys only for unchanged server IDs still missing a saved
 key, and resets removal confirmation.
 
 Submitting the same preview again returns the same collection, including after
 a lost response or an application restart. Reusing the collection with a new
 preview recomputes the shortfall, including servers queued by earlier launches.
-It does not add another full copy of the lineup. If the shortfall or assigned
-capacity changes before submission, the request is rejected without changes and
+It does not add another full copy of the lineup. If the shortfall or reviewed
+identities change before submission, the request is rejected without changes and
 requires a fresh preview. Previously saved additive previews are checked against
 the current shortfall too.
 
@@ -101,7 +109,7 @@ Only then can the next queued server start. Set **Concurrent server startups** i
 a positive integer to allow more simultaneous startups. Each worker pass admits
 at most the available slots, even if an attempt fails immediately.
 
-Names, ports, and queue order survive application restarts. A server still booting
+Names, any assigned ports, and queue order survive application restarts. A server still booting
 or waiting for health keeps its slot until it passes verification or is removed.
 A lost startup response also keeps the slot because the container may already be
 running. A confirmed creation failure is recorded for review, allowing later
@@ -125,7 +133,7 @@ remain visible in Provisioning activity and history.
 5. Choose **Launch server** and follow the operation to verified health.
 
 A preview lasts 30 minutes. It neither reserves capacity nor saves a key. If
-another launch takes the lobby or port, submission stops before saving the key;
+another launch takes the lobby, submission stops before saving the key;
 refresh the preview and review the newly assigned ID. Refreshing a preview clears
 the typed key so it cannot accidentally be saved against a different lobby.
 
@@ -161,14 +169,15 @@ without a recorded recipe retain their history but cannot offer this action.
 
 A preview shows the currently promoted managed files, configuration, and image
 reference for the selected map. It does not pin those versions. New launches
-resolve the promoted setup again when submitted and when provisioning begins, so
+resolve the promoted setup again when provisioning begins, so
 promotions made after a preview or while a job is queued apply to that server.
-This applies to collection collections, single-server launches, and **Launch again**.
+This applies to collections, single-server launches, and **Launch again**.
 
 Previews and queued jobs store file metadata without copying script or plugin
 contents for every lobby. Provisioning captures the bytes when each job starts.
-Native launch preparation reads files and encodes command payloads in a worker
-thread. The controller rechecks capacity, promotions, credentials, and safety
+Native host ranking, file reads, and command encoding run in a worker thread.
+The controller makes a short atomic reservation check after ranking, then
+rechecks capacity, promotions, credentials, and safety
 before submitting those commands. Agent delivery streams the stored payloads in
 small chunks, while progress polling reads metadata without loading file bytes.
 
@@ -180,7 +189,8 @@ also use the promoted setup at provisioning time.
 
 A submitted preview is bound to one deployment: retrying the same submission
 returns the original operation, including after an application restart or a lost
-HTTP response. The reviewed game/map, lobby ID, port, and key checks are preserved.
+HTTP response. The reviewed game/map, lobby ID, and key checks are preserved;
+native ports are reserved only during provisioning.
 Already running servers continue to receive updates through campaigns, betas,
 and drift reconciliation using the server's game/map scopes.
 Container tags can resolve to newer images; use a digest for a fixed image.

docs/native-hosts.md

@@ -34,11 +34,12 @@ Status replies still expire normally, and native timeout diagnostics are retaine
    progress and errors. Wait for verification before launching. Hosts report
    connection and readiness separately; deployment also checks the particular
    game files, image, capacity and ports required for the launch.
-6. Launch a server or collection. The host, latest game files and default
-   runtime are selected automatically after checking capacity. An advanced runtime override is
+6. Launch a server or collection. The latest game files and default runtime
+   are selected automatically. An advanced runtime override is
    available. If the latest files change after previewing, refresh the preview
    before launching.
-   EC selects lobby IDs and reserves UDP/TCP ports atomically.
+   Apply saves the requested lobby IDs. Provisioning selects a host and atomically
+   reserves UDP/TCP ports when capacity is available.
    Saved Plutonium server keys remain associated with their exact server names.
 
 A fresh destination host downloads centrally stored game files without manual
@@ -133,14 +134,19 @@ remain unknown). Scheduling checks capacity against the latest heartbeat and use
 the last five minutes to rank eligible hosts; the Inventory panel shows the latest
 reading.
 
-New servers are placed automatically. Preview and submission perform local
-capacity checks without contacting agents: a connected, active host must have
+New servers are placed automatically. Collection previews use cached availability
+estimates and submission saves unplaced jobs without running capacity checks.
+Provisioning checks each job asynchronously: a connected, active host must have
 verified game files, its runtime, storage, a free UDP/TCP port, and valid CPU and
 memory measurements. Hosts with stale or missing measurements cannot accept new
 servers. Existing API clients may still explicitly constrain a launch to a host.
 
-Admission reserves a provisional host and port atomically, including collection
-launches. Agent **1.4.6** fixes container discovery for resource sampling (the
+Queued jobs retain their accepted game-file version. Automatic host preparation
+also downloads the files and runtime images needed by unplaced jobs; a newer
+upload does not remove files still needed by the queue.
+
+Only provisioning reserves a host and port. Agent **1.4.6** fixes container
+discovery for resource sampling (the
 earlier collector omitted `label=` from its Docker filter). It samples each
 managed running container with a bounded
 [`docker stats`](https://docs.docker.com/reference/cli/docker/container/stats/)
@@ -180,19 +186,19 @@ heartbeat ingestion, at most once per minute per host/fleet phase; new peaks
 raise the cache immediately. Launch admission reads only the cached model.
 
 Scheduling still leaves **5% host CPU and 500 MB RAM (500,000,000 bytes)** for the OS and bursts.
-These allowances are scheduling estimates, not Docker limits. Collection admission
-budgets new and never-submitted queued servers at their running allowance, rather
-than assuming the entire collection starts simultaneously. Current host CPU and
+These allowances are scheduling estimates, not Docker limits. Collection previews
+estimate how many new servers fit at their running allowance. Jobs may still be
+queued when that estimate is insufficient. Current host CPU and
 available memory account for existing containers and other processes. Observed
 containers are not topped up to historical startup or running peaks, even while
 awaiting health verification. Established instances are covered by host usage
 when individual container measurements are unavailable.
 
 Dispatch checks the next launch's startup allowance against live host usage.
-Other never-submitted queued launches wait their turn and retain their port
-reservations. Submitted or uncertain launches missing a matching-generation
+Unplaced queued launches wait their turn without holding physical resources.
+Any previously assigned reservations remain protected. Submitted or uncertain launches missing a matching-generation
 measurement still consume a full startup allowance until telemetry accounts for
-them. This lets collections queue based on running demand and start as capacity
+them. This lets collections start as capacity
 becomes available, without stacking every queued server's startup peak.
 Capacity errors report projected usage, usable capacity and reservations.
 
@@ -201,7 +207,9 @@ current CPU and available memory against reservations and headroom.
 Among hosts that fit, it chooses the lowest projected CPU/memory utilization,
 using the busier of the latest reading and the five-minute average for ranking.
 A historical resource dip cannot block a host whose current capacity has recovered.
-Provisioning moves the provisional reservation atomically and records the final host in the launch recipe. Host network defaults
+Historical ranking runs in a worker thread. The controller rechecks current
+capacity and reserves the selected host and port atomically, then records them
+in the launch recipe. Host network defaults
 are resolved again for the final host; explicit overrides are preserved. If no
 host qualifies, the request waits and is checked again. Once remote staging or
 creation has been submitted, placement stays fixed to prevent duplicate instances.