Queue native collection launches before async placement

This commit is contained in:
2026-10-05 11:35:50 -04:00
parent 1405114a14
commit 7c76851252
34 changed files with 1324 additions and 244 deletions
+5 -4
View File
@@ -39,10 +39,11 @@ including SQLite metadata and published game-file storage.
shows setup progress. Automatic agent updates are enabled by default. shows setup progress. Automatic agent updates are enabled by default.
3. Add the exact Plutonium server keys in **Keys**, or supply missing keys when 3. Add the exact Plutonium server keys in **Keys**, or supply missing keys when
launching. launching.
4. Launch a server or collection. EC quickly checks CPU, memory and port capacity, 4. Launch a server or collection. Collection previews show cached capacity
then automatically places each server on the least loaded eligible host. The estimates; Apply saves the queue and opens its progress page. Provisioning
latest game files and default runtime are selected automatically. EC reserves checks capacity and assigns each server a host and port when it can start.
resources for queued launches and verifies startup. CPU/memory allowances learn The latest game files and default runtime are selected automatically.
CPU/memory allowances learn
from actual container usage, using temporary defaults while calibrating. from actual container usage, using temporary defaults while calibrating.
See [Native host management](docs/native-hosts.md) for installation, credentials, See [Native host management](docs/native-hosts.md) for installation, credentials,
+30 -20
View File
@@ -29,7 +29,7 @@ It fills deficits without changing existing servers' files.
2. Add maps from Black Ops I or II and set the desired total lobbies for each. 2. Add maps from Black Ops I or II and set the desired total lobbies for each.
3. Choose **Save & preview** (or **Preview fleet changes** on a saved collection). 3. Choose **Save & preview** (or **Preview fleet changes** on a saved collection).
Choose **Fill missing servers** or **Match collection exactly**, then review Choose **Fill missing servers** or **Match collection exactly**, then review
the server IDs and game ports. the server IDs and estimated capacity. Native host ports are assigned later.
4. Lobbies with a saved key show **Saved key ready** and reuse it automatically. 4. Lobbies with a saved key show **Saved key ready** and reuse it automatically.
Enter a Plutonium key only for lobbies missing one. Update saved keys in **Keys**. Enter a Plutonium key only for lobbies missing one. Update saved keys in **Keys**.
5. Choose **Queue N servers** (or confirm removals and choose **Apply** in exact 5. Choose **Queue N servers** (or confirm removals and choose **Apply** in exact
@@ -51,17 +51,18 @@ Unstarted creations are cancelled without booting a container. Running servers
use the normal removal queue and require a fresh zero-player RCON response. use the normal removal queue and require a fresh zero-player RCON response.
Occupied or unreachable servers wait. Already pending removals are reused. Occupied or unreachable servers wait. Already pending removals are reused.
Creation reservations and removals are queued in one database transaction. A Creation intents and removals are queued in one database transaction. A
missing key or changed fleet rejects the entire request. Any fleet identity, missing key or changed fleet rejects the entire request. Any fleet identity,
map, port, or pending-removal change since an exact preview requires a new review. map, port, or pending-removal change since an exact preview requires a new review.
Servers added after submission are not included. Applying the same preview again Servers added after submission are not included. Applying the same preview again
returns the same collection without repeating operations. returns the same collection without repeating operations.
Ports belonging to reviewed removals can be reserved for new lobbies even when Native launches can be queued even when the current port range is full. Removed
the current port range is full. A new lobby waits until its port's removal has servers retain their ports until cleanup completes; provisioning assigns newly
completed; a failed, interrupted, or cancelled removal keeps that launch blocked, available ports as each creation starts. A failed or cancelled removal can leave
with the reason shown in its progress. Other launches with available capacity can creations waiting for capacity, while other eligible hosts can still be used.
proceed. Reservations and dependencies survive app restarts. The progress page Legacy launches retain their reviewed port dependencies. Queued work and any
assigned reservations survive app restarts. The progress page
tracks creations and removals, including removal-only plans, and lists the servers tracks creations and removals, including removal-only plans, and lists the servers
that were retained. The existing operation detail page offers removal cancellation that were retained. The existing operation detail page offers removal cancellation
and cleanup recovery where applicable. and cleanup recovery where applicable.
@@ -79,18 +80,25 @@ Collections saved by the earlier single-server release load as one-server lineup
and can be expanded directly. Any file snapshots in older collections are ignored and can be expanded directly. Any file snapshots in older collections are ignored
when preparing new launches. when preparing new launches.
A collection preview does not reserve capacity. At submission, all lobby IDs, A native collection preview reads a shared capacity snapshot, maintained from host
ports, and keys are validated again. The entire collection is then queued in one heartbeats and fleet changes. It estimates available ports, CPU, memory, and startup
database transaction. If any server lacks a valid key or capacity has changed, slots without running placement for each server. Missing or insufficient capacity
no jobs, reservations, or key changes from that submission are committed. Refreshing is a warning: provisioning can wait for it. The preview does not reserve resources.
Apply validates the reviewed configuration, lobby IDs, keys, and fleet changes, saves all jobs in
one database transaction, and returns the progress page ID. Native jobs initially
have no host or port; each job performs its capacity checks asynchronously during
provisioning. If a server lacks a valid key or its reviewed identity changed,
no jobs or key changes from that submission are committed. Legacy submissions also
revalidate their reviewed ports. Refreshing
the preview retains typed keys only for unchanged server IDs still missing a saved the preview retains typed keys only for unchanged server IDs still missing a saved
key, and resets removal confirmation. key, and resets removal confirmation.
Submitting the same preview again returns the same collection, including after Submitting the same preview again returns the same collection, including after
a lost response or an application restart. Reusing the collection with a new a lost response or an application restart. Reusing the collection with a new
preview recomputes the shortfall, including servers queued by earlier launches. preview recomputes the shortfall, including servers queued by earlier launches.
It does not add another full copy of the lineup. If the shortfall or assigned It does not add another full copy of the lineup. If the shortfall or reviewed
capacity changes before submission, the request is rejected without changes and identities change before submission, the request is rejected without changes and
requires a fresh preview. Previously saved additive previews are checked against requires a fresh preview. Previously saved additive previews are checked against
the current shortfall too. the current shortfall too.
@@ -101,7 +109,7 @@ Only then can the next queued server start. Set **Concurrent server startups** i
a positive integer to allow more simultaneous startups. Each worker pass admits a positive integer to allow more simultaneous startups. Each worker pass admits
at most the available slots, even if an attempt fails immediately. at most the available slots, even if an attempt fails immediately.
Names, ports, and queue order survive application restarts. A server still booting Names, any assigned ports, and queue order survive application restarts. A server still booting
or waiting for health keeps its slot until it passes verification or is removed. or waiting for health keeps its slot until it passes verification or is removed.
A lost startup response also keeps the slot because the container may already be A lost startup response also keeps the slot because the container may already be
running. A confirmed creation failure is recorded for review, allowing later running. A confirmed creation failure is recorded for review, allowing later
@@ -125,7 +133,7 @@ remain visible in Provisioning activity and history.
5. Choose **Launch server** and follow the operation to verified health. 5. Choose **Launch server** and follow the operation to verified health.
A preview lasts 30 minutes. It neither reserves capacity nor saves a key. If A preview lasts 30 minutes. It neither reserves capacity nor saves a key. If
another launch takes the lobby or port, submission stops before saving the key; another launch takes the lobby, submission stops before saving the key;
refresh the preview and review the newly assigned ID. Refreshing a preview clears refresh the preview and review the newly assigned ID. Refreshing a preview clears
the typed key so it cannot accidentally be saved against a different lobby. the typed key so it cannot accidentally be saved against a different lobby.
@@ -161,14 +169,15 @@ without a recorded recipe retain their history but cannot offer this action.
A preview shows the currently promoted managed files, configuration, and image A preview shows the currently promoted managed files, configuration, and image
reference for the selected map. It does not pin those versions. New launches reference for the selected map. It does not pin those versions. New launches
resolve the promoted setup again when submitted and when provisioning begins, so resolve the promoted setup again when provisioning begins, so
promotions made after a preview or while a job is queued apply to that server. promotions made after a preview or while a job is queued apply to that server.
This applies to collection collections, single-server launches, and **Launch again**. This applies to collections, single-server launches, and **Launch again**.
Previews and queued jobs store file metadata without copying script or plugin Previews and queued jobs store file metadata without copying script or plugin
contents for every lobby. Provisioning captures the bytes when each job starts. contents for every lobby. Provisioning captures the bytes when each job starts.
Native launch preparation reads files and encodes command payloads in a worker Native host ranking, file reads, and command encoding run in a worker thread.
thread. The controller rechecks capacity, promotions, credentials, and safety The controller makes a short atomic reservation check after ranking, then
rechecks capacity, promotions, credentials, and safety
before submitting those commands. Agent delivery streams the stored payloads in before submitting those commands. Agent delivery streams the stored payloads in
small chunks, while progress polling reads metadata without loading file bytes. small chunks, while progress polling reads metadata without loading file bytes.
@@ -180,7 +189,8 @@ also use the promoted setup at provisioning time.
A submitted preview is bound to one deployment: retrying the same submission A submitted preview is bound to one deployment: retrying the same submission
returns the original operation, including after an application restart or a lost returns the original operation, including after an application restart or a lost
HTTP response. The reviewed game/map, lobby ID, port, and key checks are preserved. HTTP response. The reviewed game/map, lobby ID, and key checks are preserved;
native ports are reserved only during provisioning.
Already running servers continue to receive updates through campaigns, betas, Already running servers continue to receive updates through campaigns, betas,
and drift reconciliation using the server's game/map scopes. and drift reconciliation using the server's game/map scopes.
Container tags can resolve to newer images; use a digest for a fixed image. Container tags can resolve to newer images; use a digest for a fixed image.
+22 -14
View File
@@ -34,11 +34,12 @@ Status replies still expire normally, and native timeout diagnostics are retaine
progress and errors. Wait for verification before launching. Hosts report progress and errors. Wait for verification before launching. Hosts report
connection and readiness separately; deployment also checks the particular connection and readiness separately; deployment also checks the particular
game files, image, capacity and ports required for the launch. game files, image, capacity and ports required for the launch.
6. Launch a server or collection. The host, latest game files and default 6. Launch a server or collection. The latest game files and default runtime
runtime are selected automatically after checking capacity. An advanced runtime override is are selected automatically. An advanced runtime override is
available. If the latest files change after previewing, refresh the preview available. If the latest files change after previewing, refresh the preview
before launching. before launching.
EC selects lobby IDs and reserves UDP/TCP ports atomically. Apply saves the requested lobby IDs. Provisioning selects a host and atomically
reserves UDP/TCP ports when capacity is available.
Saved Plutonium server keys remain associated with their exact server names. Saved Plutonium server keys remain associated with their exact server names.
A fresh destination host downloads centrally stored game files without manual A fresh destination host downloads centrally stored game files without manual
@@ -133,14 +134,19 @@ remain unknown). Scheduling checks capacity against the latest heartbeat and use
the last five minutes to rank eligible hosts; the Inventory panel shows the latest the last five minutes to rank eligible hosts; the Inventory panel shows the latest
reading. reading.
New servers are placed automatically. Preview and submission perform local New servers are placed automatically. Collection previews use cached availability
capacity checks without contacting agents: a connected, active host must have estimates and submission saves unplaced jobs without running capacity checks.
Provisioning checks each job asynchronously: a connected, active host must have
verified game files, its runtime, storage, a free UDP/TCP port, and valid CPU and verified game files, its runtime, storage, a free UDP/TCP port, and valid CPU and
memory measurements. Hosts with stale or missing measurements cannot accept new memory measurements. Hosts with stale or missing measurements cannot accept new
servers. Existing API clients may still explicitly constrain a launch to a host. servers. Existing API clients may still explicitly constrain a launch to a host.
Admission reserves a provisional host and port atomically, including collection Queued jobs retain their accepted game-file version. Automatic host preparation
launches. Agent **1.4.6** fixes container discovery for resource sampling (the also downloads the files and runtime images needed by unplaced jobs; a newer
upload does not remove files still needed by the queue.
Only provisioning reserves a host and port. Agent **1.4.6** fixes container
discovery for resource sampling (the
earlier collector omitted `label=` from its Docker filter). It samples each earlier collector omitted `label=` from its Docker filter). It samples each
managed running container with a bounded managed running container with a bounded
[`docker stats`](https://docs.docker.com/reference/cli/docker/container/stats/) [`docker stats`](https://docs.docker.com/reference/cli/docker/container/stats/)
@@ -180,19 +186,19 @@ heartbeat ingestion, at most once per minute per host/fleet phase; new peaks
raise the cache immediately. Launch admission reads only the cached model. raise the cache immediately. Launch admission reads only the cached model.
Scheduling still leaves **5% host CPU and 500 MB RAM (500,000,000 bytes)** for the OS and bursts. Scheduling still leaves **5% host CPU and 500 MB RAM (500,000,000 bytes)** for the OS and bursts.
These allowances are scheduling estimates, not Docker limits. Collection admission These allowances are scheduling estimates, not Docker limits. Collection previews
budgets new and never-submitted queued servers at their running allowance, rather estimate how many new servers fit at their running allowance. Jobs may still be
than assuming the entire collection starts simultaneously. Current host CPU and queued when that estimate is insufficient. Current host CPU and
available memory account for existing containers and other processes. Observed available memory account for existing containers and other processes. Observed
containers are not topped up to historical startup or running peaks, even while containers are not topped up to historical startup or running peaks, even while
awaiting health verification. Established instances are covered by host usage awaiting health verification. Established instances are covered by host usage
when individual container measurements are unavailable. when individual container measurements are unavailable.
Dispatch checks the next launch's startup allowance against live host usage. Dispatch checks the next launch's startup allowance against live host usage.
Other never-submitted queued launches wait their turn and retain their port Unplaced queued launches wait their turn without holding physical resources.
reservations. Submitted or uncertain launches missing a matching-generation Any previously assigned reservations remain protected. Submitted or uncertain launches missing a matching-generation
measurement still consume a full startup allowance until telemetry accounts for measurement still consume a full startup allowance until telemetry accounts for
them. This lets collections queue based on running demand and start as capacity them. This lets collections start as capacity
becomes available, without stacking every queued server's startup peak. becomes available, without stacking every queued server's startup peak.
Capacity errors report projected usage, usable capacity and reservations. Capacity errors report projected usage, usable capacity and reservations.
@@ -201,7 +207,9 @@ current CPU and available memory against reservations and headroom.
Among hosts that fit, it chooses the lowest projected CPU/memory utilization, Among hosts that fit, it chooses the lowest projected CPU/memory utilization,
using the busier of the latest reading and the five-minute average for ranking. using the busier of the latest reading and the five-minute average for ranking.
A historical resource dip cannot block a host whose current capacity has recovered. A historical resource dip cannot block a host whose current capacity has recovered.
Provisioning moves the provisional reservation atomically and records the final host in the launch recipe. Host network defaults Historical ranking runs in a worker thread. The controller rechecks current
capacity and reserves the selected host and port atomically, then records them
in the launch recipe. Host network defaults
are resolved again for the final host; explicit overrides are preserved. If no are resolved again for the final host; explicit overrides are preserved. If no
host qualifies, the request waits and is checked again. Once remote staging or host qualifies, the request waits and is checked again. Once remote staging or
creation has been submitted, placement stays fixed to prevent duplicate instances. creation has been submitted, placement stays fixed to prevent duplicate instances.
+15 -1
View File
@@ -11,7 +11,7 @@ export interface LaunchFileSummary {
} }
export interface LaunchRecipeSummary { export interface LaunchRecipeSummary {
native?: { hostId: string; libraryId: string; image?: string; bindAddress?: string; publicAddress?: string; publicPort?: number }; native?: import('./hosts.js').NativePlacement;
game: ZombieGame; game: ZombieGame;
mapSlug: string; mapSlug: string;
mapName: string; mapName: string;
@@ -35,6 +35,18 @@ export interface BlueprintSummary {
export type CollectionMode = 'fill' | 'exact'; export type CollectionMode = 'fill' | 'exact';
export interface LaunchAvailability {
updatedAt: string | null;
requested: number;
estimatedAvailable: number | null;
portsAvailable: number | null;
cpuCoresAvailable: number | null;
memoryBytesAvailable: number | null;
startupSlots: number | null;
startupLimit: number | null;
queued: number | null;
}
export interface CollectionPreview { export interface CollectionPreview {
id: string; id: string;
expiresAt: string; expiresAt: string;
@@ -47,6 +59,8 @@ export interface CollectionPreview {
coverage: { name: string; health: string; href: string }[]; recipe: LaunchRecipeSummary }[]; coverage: { name: string; health: string; href: string }[]; recipe: LaunchRecipeSummary }[];
servers: { native?: import('./hosts.js').NativePlacement; entryIndex: number; name: string | null; port: number | null; hasSavedKey: boolean }[]; servers: { native?: import('./hosts.js').NativePlacement; entryIndex: number; name: string | null; port: number | null; hasSavedKey: boolean }[];
blockers: string[]; blockers: string[];
warnings?: string[];
availability?: LaunchAvailability;
ready: boolean; ready: boolean;
} }
+2 -2
View File
@@ -116,9 +116,9 @@ export interface NativeDeploymentSelection {
bindAddress?: string; publicAddress?: string; publicPort?: number; bindAddress?: string; publicAddress?: string; publicPort?: number;
} }
/** The host/port are provisional until the creation command is submitted. */ /** Queued automatic launches have no assigned host or port until provisioning. */
export interface NativePlacement { export interface NativePlacement {
hostId: string; libraryId: string; image?: string; hostId?: string; libraryId: string; image?: string;
bindAddress?: string; publicAddress?: string; publicPort?: number; bindAddress?: string; publicAddress?: string; publicPort?: number;
automatic?: boolean; automatic?: boolean;
network?: { bindAddress?: string; publicAddress?: string; publicPort?: number }; network?: { bindAddress?: string; publicAddress?: string; publicPort?: number };
+13 -1
View File
@@ -15,7 +15,19 @@ export function nativeInstance(server: string): NativeInstance | null {
} }
export function nativeDeployment(id: string): boolean { export function nativeDeployment(id: string): boolean {
return !!db().prepare('SELECT 1 FROM native_provisioning WHERE deployment_id = ?').get(id); return !!db().prepare(`SELECT 1 FROM native_provisioning WHERE deployment_id = ?
UNION ALL SELECT 1 FROM native_creation_intents WHERE deployment_id = ?
UNION ALL SELECT 1 FROM native_creation_cancellations c
JOIN native_creation_intents i ON i.deployment_id=c.creation_id WHERE c.cleanup_id = ? LIMIT 1`).get(id, id, id);
}
export function unplacedNativeCreation(server: string): string | null {
const row = db().prepare(`SELECT d.id FROM server_deployments d
JOIN native_creation_intents i ON i.deployment_id=d.id
LEFT JOIN native_provisioning n ON n.deployment_id=d.id
WHERE d.server=? AND d.server_deleted_at IS NULL AND d.action='create' AND d.state='queued'
AND n.deployment_id IS NULL ORDER BY d.rowid DESC LIMIT 1`).get(server) as { id: string } | undefined;
return row?.id ?? null;
} }
const observations = new Map<string, Promise<unknown>>(); const observations = new Map<string, Promise<unknown>>();
+29
View File
@@ -1,7 +1,12 @@
import type Database from 'better-sqlite3'; import type Database from 'better-sqlite3';
export function migrateNativeInstances(handle: Database.Database): void { export function migrateNativeInstances(handle: Database.Database): void {
migrateUnplacedCreationPort(handle);
handle.exec(` handle.exec(`
CREATE TABLE IF NOT EXISTS native_creation_intents (
deployment_id TEXT PRIMARY KEY REFERENCES server_deployments(id) ON DELETE CASCADE,
selection TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS native_instances ( CREATE TABLE IF NOT EXISTS native_instances (
id TEXT PRIMARY KEY, server TEXT NOT NULL, host_id TEXT NOT NULL, id TEXT PRIMARY KEY, server TEXT NOT NULL, host_id TEXT NOT NULL,
generation INTEGER NOT NULL DEFAULT 1, library_id TEXT NOT NULL, generation INTEGER NOT NULL DEFAULT 1, library_id TEXT NOT NULL,
@@ -24,3 +29,27 @@ export function migrateNativeInstances(handle: Database.Database): void {
); );
`); `);
} }
/** Accepted native creations receive their host port only when provisioning places them. */
function migrateUnplacedCreationPort(handle: Database.Database): void {
const columns = handle.prepare('PRAGMA table_info(server_creation_jobs)').all() as { name: string; notnull: number }[];
if (!columns.find(column => column.name === 'port')?.notnull) return;
const { sql } = handle.prepare("SELECT sql FROM sqlite_master WHERE type='table' AND name='server_creation_jobs'").get() as { sql: string };
const replacement = sql.replace(/\bport\s+INTEGER\s+NOT\s+NULL\b/i, 'port INTEGER');
if (replacement === sql) throw new Error('Could not migrate unplaced creation ports.');
const dependent = handle.prepare("SELECT sql FROM sqlite_master WHERE tbl_name='server_creation_jobs' AND type IN ('index','trigger') AND sql IS NOT NULL")
.all() as { sql: string }[];
const foreignKeys = handle.pragma('foreign_keys', { simple: true });
handle.pragma('foreign_keys = OFF');
try {
handle.transaction(() => {
handle.exec(replacement.replace('server_creation_jobs', 'server_creation_jobs_unplaced'));
const names = columns.map(column => `"${column.name.replaceAll('"', '""')}"`).join(', ');
handle.exec(`INSERT INTO server_creation_jobs_unplaced(rowid,${names}) SELECT rowid,${names} FROM server_creation_jobs;
DROP TABLE server_creation_jobs;
ALTER TABLE server_creation_jobs_unplaced RENAME TO server_creation_jobs;`);
for (const object of dependent) handle.exec(object.sql);
if ((handle.pragma('foreign_key_check') as unknown[]).length) throw new Error('Unplaced creation migration found broken references.');
})();
} finally { handle.pragma(`foreign_keys = ${foreignKeys ? 'ON' : 'OFF'}`); }
}
+1 -1
View File
@@ -475,7 +475,7 @@ CREATE INDEX IF NOT EXISTS idx_server_deployment_events ON server_deployment_eve
CREATE TABLE IF NOT EXISTS server_creation_jobs ( CREATE TABLE IF NOT EXISTS server_creation_jobs (
deployment_id TEXT PRIMARY KEY REFERENCES server_deployments(id) ON DELETE CASCADE, deployment_id TEXT PRIMARY KEY REFERENCES server_deployments(id) ON DELETE CASCADE,
server TEXT NOT NULL, server TEXT NOT NULL,
port INTEGER NOT NULL, port INTEGER,
game_mode TEXT NOT NULL, game_mode TEXT NOT NULL,
map_slug TEXT NOT NULL, map_slug TEXT NOT NULL,
expected_map TEXT NOT NULL expected_map TEXT NOT NULL
+76 -47
View File
@@ -4,18 +4,18 @@ import type { CollectionDetail, CollectionMode, CollectionPreview, CollectionSum
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import { config, komodoConfigured, rconConfigured } from '../config.js'; import { config, komodoConfigured, rconConfigured } from '../config.js';
import { blueprintEntries, summarizeRecipe, type BlueprintEntry } from './launch-recipes.js'; import { blueprintEntries, summarizeRecipe, type BlueprintEntry } from './launch-recipes.js';
import { creationQueue, destroyServer, nextAvailablePort, nextLobbyNumber, queueCreateServer } from '../provisioning/index.js'; import { creationQueue, destroyServer, nextAvailablePort, queueCreateServer } from '../provisioning/index.js';
import { deploymentStage, finishDeployment, recordDeploymentEvent } from './deployments.js'; import { deploymentStage, finishDeployment, recordDeploymentEvent } from './deployments.js';
import { housePartyTargets } from './house-party.js'; import { housePartyTargets } from './house-party.js';
import { getServerKey } from './server-keys.js';
import { serverObservation } from './status.js'; import { serverObservation } from './status.js';
import { activeBetaRemovalBlocker, getServer } from './servers.js'; import { activeBetaRemovalBlocker, getServer } from './servers.js';
import { parseServerName, zombieMapBySlug, zombieMaps } from './map-aliases.js'; import { parseServerName, zombieMapBySlug, zombieMaps } from './map-aliases.js';
import { nativeSelection, nextNativePort } from '../provisioning/native.js'; import { nativeLaunchSelection, queueNativeCreationIntent } from '../provisioning/native.js';
import { listHosts } from '../hosts/index.js'; import { listHosts } from '../hosts/index.js';
import { nativeInstance } from '../backends/native.js'; import { nativeDeployment, nativeInstance } from '../backends/native.js';
import { launchAvailability } from '../hosts/availability.js';
function coverage(entries: BlueprintEntry[]) { function coverage(entries: BlueprintEntry[], includeHealth = true) {
type Server = { name: string; game: string | null; map_slug: string | null; map: string | null }; type Server = { name: string; game: string | null; map_slug: string | null; map: string | null };
const registered = db().prepare('SELECT name, game, map_slug, map FROM servers WHERE destroyed_at IS NULL').all() as Server[]; const registered = db().prepare('SELECT name, game, map_slug, map FROM servers WHERE destroyed_at IS NULL').all() as Server[];
const queued = db().prepare(`SELECT j.server AS name, substr(j.server, 1, 2) AS game, j.map_slug, j.expected_map AS map const queued = db().prepare(`SELECT j.server AS name, substr(j.server, 1, 2) AS game, j.map_slug, j.expected_map AS map
@@ -37,14 +37,14 @@ function coverage(entries: BlueprintEntry[]) {
const matches = zombieMaps(game).filter(map => map.canonical === server.map); const matches = zombieMaps(game).filter(map => map.canonical === server.map);
return matches.length === 1 && matches[0].slug === entry.recipe.mapSlug; return matches.length === 1 && matches[0].slug === entry.recipe.mapSlug;
}).map(server => server.name).sort(); }).map(server => server.name).sort();
const coverage = existingServers.map(name => { const coverage = includeHealth ? existingServers.map(name => {
const operation = db().prepare("SELECT id, state FROM server_deployments WHERE server = ? AND server_deleted_at IS NULL AND action IN ('create','rebuild') AND state IN ('queued', 'running', 'awaiting-health') ORDER BY rowid DESC LIMIT 1").get(name) as { id: string; state: string } | undefined; const operation = db().prepare("SELECT id, state FROM server_deployments WHERE server = ? AND server_deleted_at IS NULL AND action IN ('create','rebuild') AND state IN ('queued', 'running', 'awaiting-health') ORDER BY rowid DESC LIMIT 1").get(name) as { id: string; state: string } | undefined;
const server = getServer(name); const server = getServer(name);
return { name, health: operation ? operation.state === 'queued' ? 'queued' : 'starting' return { name, health: operation ? operation.state === 'queued' ? 'queued' : 'starting'
: server?.container_state && ['stopped', 'exited', 'created', 'dead'].includes(server.container_state) ? 'stopped' : server?.container_state && ['stopped', 'exited', 'created', 'dead'].includes(server.container_state) ? 'stopped'
: server ? serverObservation(server).health : 'offline', : server ? serverObservation(server).health : 'offline',
href: operation ? `/provisioning/${encodeURIComponent(operation.id)}` : `/servers/${encodeURIComponent(name)}` }; href: operation ? `/provisioning/${encodeURIComponent(operation.id)}` : `/servers/${encodeURIComponent(name)}` };
}); }) : [];
return { existingServers, coverage, existing: existingServers.length, missing: Math.max(0, entry.count - existingServers.length) }; return { existingServers, coverage, existing: existingServers.length, missing: Math.max(0, entry.count - existingServers.length) };
}); });
} }
@@ -57,8 +57,8 @@ function fleetSnapshot() {
})); }));
} }
function collectionPlan(entries: BlueprintEntry[], mode: CollectionMode) { function collectionPlan(entries: BlueprintEntry[], mode: CollectionMode, includeHealth = true) {
const counts = coverage(entries); const counts = coverage(entries, includeHealth);
// Preserve deployed lobbies before queued ones, then the lowest lobby IDs. // Preserve deployed lobbies before queued ones, then the lowest lobby IDs.
const retainedServers = counts.flatMap((entry, index) => [...entry.existingServers].sort((a, b) => const retainedServers = counts.flatMap((entry, index) => [...entry.existingServers].sort((a, b) =>
Number(!!getServer(b)) - Number(!!getServer(a)) || a.localeCompare(b, 'en', { numeric: true }) Number(!!getServer(b)) - Number(!!getServer(a)) || a.localeCompare(b, 'en', { numeric: true })
@@ -71,50 +71,56 @@ function collectionPlan(entries: BlueprintEntry[], mode: CollectionMode) {
return { counts, retainedServers, removals }; return { counts, retainedServers, removals };
} }
function allocate(entries: BlueprintEntry[], releasingServers: string[] = []): CollectionPreview['servers'] { function allocate(entries: BlueprintEntry[], releasingServers: string[] = [], counts = coverage(entries)): CollectionPreview['servers'] {
const names: string[] = []; const occupied = db().prepare(`SELECT name FROM servers UNION
SELECT j.server AS name FROM server_creation_jobs j JOIN server_deployments d ON d.id=j.deployment_id
WHERE d.server_deleted_at IS NULL AND (d.state IN ('queued','running','awaiting-health')
OR (d.action='rebuild' AND (d.state='waiting' OR (d.state IN ('failed','interrupted') AND d.stage='Removing server files and registry'))))`)
.all() as { name: string }[];
const lobbies = new Map<string, Set<number>>();
for (const { name } of occupied) {
const match = /^(.*)_(\d+)$/.exec(name);
if (!match) continue;
const taken = lobbies.get(match[1]) ?? new Set<number>();
taken.add(Number(match[2])); lobbies.set(match[1], taken);
}
const savedKeys = new Set((db().prepare("SELECT server_id FROM server_keys WHERE length(trim(pluto_key))>0").all() as { server_id: string }[]).map(row => row.server_id));
const ports: number[] = []; const ports: number[] = [];
const planned: { hostId: string; port: number }[] = [];
const counts = coverage(entries);
return entries.flatMap((entry, entryIndex) => Array.from({ length: counts[entryIndex].missing }, () => { return entries.flatMap((entry, entryIndex) => Array.from({ length: counts[entryIndex].missing }, () => {
const base = `${entry.recipe.game}_zombie_${entry.recipe.mapSlug}`; const base = `${entry.recipe.game}_zombie_${entry.recipe.mapSlug}`;
const lobby = nextLobbyNumber(base, names); const taken = lobbies.get(base) ?? new Set<number>();
const lobby = Array.from({ length: 20 }, (_, index) => index + 1).find(number => !taken.has(number)) ?? null;
const name = lobby === null ? null : `${base}_${lobby}`; const name = lobby === null ? null : `${base}_${lobby}`;
let native = entry.recipe.native; const native = entry.recipe.native;
let port: number | null; const port = native ? null : nextAvailablePort(ports, releasingServers);
if (native) { if (lobby !== null) { taken.add(lobby); lobbies.set(base, taken); }
try {
native = nativeSelection(native as unknown as Record<string, unknown>, entry.recipe.game, { planned });
port = nextNativePort(native, planned.filter(item => item.hostId === native!.hostId).map(item => item.port));
if (port !== null) planned.push({ hostId: native.hostId, port });
} catch { port = null; }
} else port = nextAvailablePort(ports, releasingServers);
if (name) names.push(name);
if (port !== null) ports.push(port); if (port !== null) ports.push(port);
return { entryIndex, name, port, ...(native ? { native } : {}), hasSavedKey: !!name && !!getServerKey(name) }; return { entryIndex, name, port, ...(native ? { native } : {}), hasSavedKey: !!name && savedKeys.has(name) };
})); }));
} }
function blockersFor(servers: CollectionPreview['servers'], entries: BlueprintEntry[] = []): string[] { function blockersFor(servers: CollectionPreview['servers'], entries: BlueprintEntry[] = []): string[] {
const blockers: string[] = []; const blockers: string[] = [];
if (!servers.length) return blockers; if (!servers.length) return blockers;
if (entries.some(entry => entry.recipe.native)) { const legacy = servers.filter(server => !server.native && !entries[server.entryIndex]?.recipe.native);
for (const entry of entries) { if (legacy.length) {
try { nativeSelection(entry.recipe.native as unknown as Record<string, unknown>, entry.recipe.game); }
catch (error) { blockers.push((error as Error).message); }
}
} else {
if (!komodoConfigured() || !config.komodo.targetServer) blockers.push('Select and prepare a host before launching.'); if (!komodoConfigured() || !config.komodo.targetServer) blockers.push('Select and prepare a host before launching.');
if (!rconConfigured()) blockers.push('Configure RCON_PASSWORD so every new server can be verified.'); if (!rconConfigured()) blockers.push('Configure RCON_PASSWORD so every new server can be verified.');
} }
if (servers.some(server => !server.name)) blockers.push('There are not enough free lobbies for one or more maps (20 per map). Reduce their counts or free capacity.'); if (servers.some(server => !server.name)) blockers.push('There are not enough free lobbies for one or more maps (20 per map). Reduce their counts or free capacity.');
if (servers.some(server => server.port === null)) blockers.push(entries.some(entry => entry.recipe.native) ? 'There is not enough host CPU, memory or port capacity for the whole collection.' : `There are not enough free game ports in ${config.portRange.min}–${config.portRange.max} for the whole collection.`); if (legacy.some(server => server.port === null)) blockers.push(`There are not enough free game ports in ${config.portRange.min}–${config.portRange.max} for the whole collection.`);
return blockers; return blockers;
} }
function removalBlockersFor(removals: CollectionPreview['removals']): string[] { function removalBlockersFor(removals: CollectionPreview['removals']): string[] {
const blockers: string[] = []; const blockers: string[] = [];
if (removals.some(server => !nativeInstance(server.name)) && (!komodoConfigured() || !rconConfigured())) blockers.push('Configure the legacy backend before removing its servers.'); const native = (name: string) => {
if (nativeInstance(name)) return true;
const creation = db().prepare(`SELECT id FROM server_deployments WHERE server=? AND server_deleted_at IS NULL
AND action='create' AND state IN ('queued','running','awaiting-health') ORDER BY rowid DESC LIMIT 1`).get(name) as { id: string } | undefined;
return !!creation && nativeDeployment(creation.id);
};
if (removals.some(server => !native(server.name)) && (!komodoConfigured() || !rconConfigured())) blockers.push('Configure the legacy backend before removing its servers.');
for (const server of removals) { for (const server of removals) {
const blocker = activeBetaRemovalBlocker(server.name); const blocker = activeBetaRemovalBlocker(server.name);
if (blocker) blockers.push(blocker); if (blocker) blockers.push(blocker);
@@ -130,24 +136,26 @@ export function previewCollection(blueprintId: unknown, mode: unknown = 'fill',
if (selection?.hostId !== undefined || !config.komodo.address || listHosts().length > 0) { if (selection?.hostId !== undefined || !config.komodo.address || listHosts().length > 0) {
const libraries = selection?.libraryIds as Record<string, unknown> | undefined; const libraries = selection?.libraryIds as Record<string, unknown> | undefined;
for (const entry of entries) { for (const entry of entries) {
entry.recipe.native = nativeSelection({ ...selection, libraryId: libraries?.[entry.recipe.game] ?? selection?.libraryId }, entry.recipe.game); entry.recipe.native = nativeLaunchSelection({ ...selection, libraryId: libraries?.[entry.recipe.game] ?? selection?.libraryId }, entry.recipe.game);
entry.recipe.image = entry.recipe.native.image!; entry.recipe.image = entry.recipe.native.image!;
} }
} }
const { counts, retainedServers, removals } = collectionPlan(entries, mode); const { counts, retainedServers, removals } = collectionPlan(entries, mode);
const servers = allocate(entries, removals.map(server => server.name)); const servers = allocate(entries, removals.map(server => server.name), counts);
const blockers = [...blockersFor(servers, entries), ...removalBlockersFor(removals)]; const blockers = [...blockersFor(servers, entries), ...removalBlockersFor(removals)];
const selections = servers.flatMap(server => server.native ? [server.native] : []);
const availability = selections.length ? launchAvailability(selections) : {};
const id = randomUUID(); const id = randomUUID();
const expiresAt = new Date(Date.now() + 30 * 60_000).toISOString(); const expiresAt = new Date(Date.now() + 30 * 60_000).toISOString();
db().prepare('DELETE FROM server_collection_previews WHERE expires_at < ? AND collection_id IS NULL').run(new Date().toISOString()); db().prepare('DELETE FROM server_collection_previews WHERE expires_at < ? AND collection_id IS NULL').run(new Date().toISOString());
db().prepare('INSERT INTO server_collection_previews (id, expires_at, blueprint_name, entries, servers, mode, fleet_snapshot) VALUES (?, ?, ?, ?, ?, ?, ?)') db().prepare('INSERT INTO server_collection_previews (id, expires_at, blueprint_name, entries, servers, mode, fleet_snapshot) VALUES (?, ?, ?, ?, ?, ?, ?)')
.run(id, expiresAt, blueprintName, JSON.stringify(entries), JSON.stringify(servers), mode, mode === 'exact' ? JSON.stringify(fleetSnapshot()) : null); .run(id, expiresAt, blueprintName, JSON.stringify(entries), JSON.stringify(servers), mode, mode === 'exact' ? JSON.stringify(fleetSnapshot()) : null);
return { id, expiresAt, blueprintName, mode, retainedServers, removals, launchConcurrency: fleetSettings().createConcurrency, servers, blockers, ready: blockers.length === 0, return { id, expiresAt, blueprintName, mode, retainedServers, removals, launchConcurrency: fleetSettings().createConcurrency, servers, blockers, ...availability, ready: blockers.length === 0,
entries: entries.map((entry, index) => ({ count: entry.count, ...counts[index], recipe: summarizeRecipe(entry.recipe) })) }; entries: entries.map((entry, index) => ({ count: entry.count, ...counts[index], recipe: summarizeRecipe(entry.recipe) })) };
})(); })();
} }
/** Reserve and enqueue the entire collection atomically. No external operations here. */ /** Save all reviewed identities and keys atomically; native placement belongs to provisioning. */
export function queueCollection(input: { previewId?: unknown; keys?: unknown }): { ok: true; collectionId: string | null } { export function queueCollection(input: { previewId?: unknown; keys?: unknown }): { ok: true; collectionId: string | null } {
if (typeof input.previewId !== 'string' || !input.previewId) throw new Error('Review a collection preview before launching.'); if (typeof input.previewId !== 'string' || !input.previewId) throw new Error('Review a collection preview before launching.');
if (input.keys !== undefined && (!input.keys || typeof input.keys !== 'object' || Array.isArray(input.keys))) throw new Error('Provide keys indexed by the exact server ID.'); if (input.keys !== undefined && (!input.keys || typeof input.keys !== 'object' || Array.isArray(input.keys))) throw new Error('Provide keys indexed by the exact server ID.');
@@ -160,19 +168,35 @@ export function queueCollection(input: { previewId?: unknown; keys?: unknown }):
if (preview.collection_id) return { ok: true as const, collectionId: preview.collection_id }; if (preview.collection_id) return { ok: true as const, collectionId: preview.collection_id };
if (preview.expires_at <= new Date().toISOString()) throw new Error('Collection preview expired. Refresh it before launching.'); if (preview.expires_at <= new Date().toISOString()) throw new Error('Collection preview expired. Refresh it before launching.');
const entries: BlueprintEntry[] = JSON.parse(preview.entries); const entries: BlueprintEntry[] = JSON.parse(preview.entries);
// Older previews embedded plugin bytes. Child previews need only the // Older previews embedded plugin bytes. Queued jobs need only the
// reviewed metadata; provisioning captures the current files at dispatch. // reviewed metadata; provisioning captures the current files at dispatch.
for (const entry of entries) entry.recipe.files = entry.recipe.files.map(({ content: _content, ...file }) => file); for (const entry of entries) entry.recipe.files = entry.recipe.files.map(({ content: _content, ...file }) => file);
const reviewed: CollectionPreview['servers'] = JSON.parse(preview.servers); const reviewed: CollectionPreview['servers'] = JSON.parse(preview.servers);
if (preview.mode === 'exact' && preview.fleet_snapshot !== JSON.stringify(fleetSnapshot())) { if (preview.mode === 'exact' && preview.fleet_snapshot !== JSON.stringify(fleetSnapshot())) {
throw new Error('The fleet changed. Nothing was queued. Refresh the preview and review the servers to keep and remove.'); throw new Error('The fleet changed. Nothing was queued. Refresh the preview and review the servers to keep and remove.');
} }
const { retainedServers, removals } = collectionPlan(entries, preview.mode); const { counts, retainedServers, removals } = collectionPlan(entries, preview.mode, false);
const releasingServers = removals.map(server => server.name); const releasingServers = removals.map(server => server.name);
const servers = allocate(entries, releasingServers); const nativeReview = reviewed.every(server => server.native || entries[server.entryIndex]?.recipe.native);
// Native Apply accepts the reviewed logical identities. Capacity and
// physical ports can change while this request waits in the queue.
const servers = nativeReview ? reviewed : allocate(entries, releasingServers, counts);
const validatedSelections = new Set<string>();
for (const server of servers) {
const recipe = entries[server.entryIndex].recipe;
const selection = server.native ?? recipe.native;
if (!selection) continue;
const identity = JSON.stringify([recipe.game, selection]);
if (validatedSelections.has(identity)) continue;
// Recheck saved configuration once per selection without allocating
// physical capacity or changing the names and files already reviewed.
nativeLaunchSelection({ ...selection }, recipe.game);
validatedSelections.add(identity);
}
const blockers = [...blockersFor(servers, entries), ...removalBlockersFor(removals)]; const blockers = [...blockersFor(servers, entries), ...removalBlockersFor(removals)];
if (blockers.length) throw new Error(blockers.join(' ')); if (blockers.length) throw new Error(blockers.join(' '));
if (servers.length !== reviewed.length || servers.some((server, i) => server.entryIndex !== reviewed[i]?.entryIndex || server.name !== reviewed[i]?.name || (!server.native?.automatic && server.port !== reviewed[i]?.port))) { if (entries.some((_entry, index) => counts[index].missing !== reviewed.filter(server => server.entryIndex === index).length)
|| servers.length !== reviewed.length || servers.some((server, i) => server.entryIndex !== reviewed[i]?.entryIndex || server.name !== reviewed[i]?.name || (!server.native && server.port !== reviewed[i]?.port))) {
throw new Error('Fleet counts, lobby or port availability changed. Nothing was queued. Refresh the preview and review each server key.'); throw new Error('Fleet counts, lobby or port availability changed. Nothing was queued. Refresh the preview and review each server key.');
} }
const expected = new Set(servers.map(server => server.name)); const expected = new Set(servers.map(server => server.name));
@@ -210,16 +234,21 @@ export function queueCollection(input: { previewId?: unknown; keys?: unknown }):
}); });
servers.forEach((server, index) => { servers.forEach((server, index) => {
const recipe = { ...entries[server.entryIndex].recipe, ...(server.native ? { native: server.native } : {}) }; const recipe = { ...entries[server.entryIndex].recipe, ...(server.native ? { native: server.native } : {}) };
const childPreviewId = randomUUID(); let result;
db().prepare('INSERT INTO server_launch_previews (id, expires_at, server, port, recipe, blueprint_name) VALUES (?, ?, ?, ?, ?, ?)') if (recipe.native) result = queueNativeCreationIntent({ server: server.name!, game: recipe.game, mapSlug: recipe.mapSlug,
.run(childPreviewId, preview.expires_at, server.name, server.port, JSON.stringify(recipe), preview.blueprint_name); selection: recipe.native, recipe, plutoKey: keys[server.name!] as string | undefined, createdBy: 'ui', blueprintName: preview.blueprint_name });
const result = queueCreateServer({ game: recipe.game, gameMode: 'zombie', mapSlug: recipe.mapSlug, else {
expectedName: server.name!, previewId: childPreviewId, plutoKey: keys[server.name!] as string | undefined, createdBy: 'ui' }, releasingServers); const childPreviewId = randomUUID();
db().prepare('INSERT INTO server_launch_previews (id, expires_at, server, port, recipe, blueprint_name) VALUES (?, ?, ?, ?, ?, ?)')
.run(childPreviewId, preview.expires_at, server.name, server.port, JSON.stringify(recipe), preview.blueprint_name);
result = queueCreateServer({ game: recipe.game, gameMode: 'zombie', mapSlug: recipe.mapSlug,
expectedName: server.name!, previewId: childPreviewId, plutoKey: keys[server.name!] as string | undefined, createdBy: 'ui' }, releasingServers);
}
// Throw so the outer transaction also reverses all earlier keys and children. // Throw so the outer transaction also reverses all earlier keys and children.
if (!result.ok) throw new Error(`${server.name}: ${result.message} Nothing was queued.`); if (!result.ok) throw new Error(`${server.name}: ${result.message} Nothing was queued.`);
db().prepare('INSERT INTO server_collection_members (collection_id, deployment_id, position) VALUES (?, ?, ?)') db().prepare('INSERT INTO server_collection_members (collection_id, deployment_id, position) VALUES (?, ?, ?)')
.run(collectionId, result.deploymentId, index); .run(collectionId, result.deploymentId, index);
for (const removal of removalJobs.filter(removal => removal.ports.includes(server.port))) { for (const removal of recipe.native ? [] : removalJobs.filter(removal => removal.ports.includes(server.port))) {
db().prepare('INSERT INTO server_creation_dependencies (deployment_id, removal_id) VALUES (?, ?)').run(result.deploymentId, removal.id); db().prepare('INSERT INTO server_creation_dependencies (deployment_id, removal_id) VALUES (?, ?)').run(result.deploymentId, removal.id);
} }
}); });
+12
View File
@@ -439,6 +439,18 @@ function libraryInUse(versionId: string, hostId?: string): boolean {
if (!handle.prepare("SELECT 1 FROM sqlite_master WHERE type='table' AND name='native_instances'").get()) return true; if (!handle.prepare("SELECT 1 FROM sqlite_master WHERE type='table' AND name='native_instances'").get()) return true;
if (handle.prepare(`SELECT 1 FROM native_instances WHERE library_id=? AND retired_at IS NULL ${hostId ? 'AND host_id=?' : ''} LIMIT 1`) if (handle.prepare(`SELECT 1 FROM native_instances WHERE library_id=? AND retired_at IS NULL ${hostId ? 'AND host_id=?' : ''} LIMIT 1`)
.get(versionId, ...(hostId ? [hostId] : []))) return true; .get(versionId, ...(hostId ? [hostId] : []))) return true;
// Accepted jobs pin their game files before placement creates an instance.
// Automatic intents may still choose any host, so retain each available copy.
if (handle.prepare("SELECT 1 FROM sqlite_master WHERE type='table' AND name='native_creation_intents'").get()
&& handle.prepare(`SELECT 1 FROM server_deployments d
JOIN native_creation_intents q ON q.deployment_id=d.id
LEFT JOIN native_provisioning n ON n.deployment_id=d.id
WHERE d.action='create' AND d.server_deleted_at IS NULL
AND d.state IN ('queued','waiting','running','awaiting-health','interrupted') AND n.deployment_id IS NULL
AND json_extract(q.selection,'$.libraryId')=?
${hostId ? `AND (json_extract(q.selection,'$.automatic')=1 OR json_extract(q.selection,'$.hostId') IS NULL
OR json_extract(q.selection,'$.hostId')=?)` : ''} LIMIT 1`)
.get(versionId, ...(hostId ? [hostId] : []))) return true;
return !!handle.prepare(`SELECT 1 FROM ec_host_operations WHERE state IN ('pending','accepted','running','unknown') return !!handle.prepare(`SELECT 1 FROM ec_host_operations WHERE state IN ('pending','accepted','running','unknown')
AND (json_extract(payload,'$.libraryVersionId')=? OR json_extract(payload,'$.libraryId')=? OR json_extract(payload,'$.importId')=?) AND (json_extract(payload,'$.libraryVersionId')=? OR json_extract(payload,'$.libraryId')=? OR json_extract(payload,'$.importId')=?)
${hostId ? 'AND host_id=?' : ''} LIMIT 1`).get(versionId, versionId, versionId, ...(hostId ? [hostId] : [])); ${hostId ? 'AND host_id=?' : ''} LIMIT 1`).get(versionId, versionId, versionId, ...(hostId ? [hostId] : []));
+5
View File
@@ -3,6 +3,11 @@ import type Database from 'better-sqlite3';
const initialized = new WeakSet<Database.Database>(); const initialized = new WeakSet<Database.Database>();
export function ensureGameFileSchema(handle: Database.Database): void { export function ensureGameFileSchema(handle: Database.Database): void {
if (initialized.has(handle)) return; if (initialized.has(handle)) return;
// Worker readers open only after the controller has initialized the schema.
if (handle.readonly) {
initialized.add(handle);
return;
}
handle.exec(` handle.exec(`
CREATE TABLE IF NOT EXISTS game_file_libraries ( CREATE TABLE IF NOT EXISTS game_file_libraries (
id TEXT PRIMARY KEY, name TEXT NOT NULL COLLATE NOCASE UNIQUE, id TEXT PRIMARY KEY, name TEXT NOT NULL COLLATE NOCASE UNIQUE,
+141
View File
@@ -0,0 +1,141 @@
import { db } from '../db/index.js';
import { fleetSettings } from '../settings.js';
import { subscribe } from '../pipeline/bus.js';
import { capacityCalibration } from './calibration.js';
import { HEARTBEAT_MAX_AGE_MS, listHosts } from './index.js';
import type { Host, NativePlacement } from '../../hosts.js';
import type { LaunchAvailability } from '../../deployment-plan.js';
interface HostAvailability {
host: Host;
freeCpu: number | null;
freeMemory: number | null;
runningCpu: number;
runningMemory: number;
ports: { port: number; bind_address: string }[];
}
interface Snapshot {
updatedAt: string;
hosts: Map<string, HostAvailability>;
startupLimit: number;
active: number;
queued: number;
}
const snapshots = new WeakMap<ReturnType<typeof db>, Snapshot>();
const scheduled = new WeakSet<ReturnType<typeof db>>();
function snapshotSchemaReady(handle: ReturnType<typeof db>): boolean {
// Host enrollment and game-file maintenance can run before provisioning is
// initialized. Missing queue data means unknown capacity, not an empty fleet.
const { count } = handle.prepare(`SELECT COUNT(*) AS count FROM sqlite_master WHERE type='table'
AND name IN ('native_instances','native_provisioning','server_deployments','settings')`).get() as { count: number };
if (count === 4) return true;
snapshots.delete(handle);
return false;
}
function queueSnapshot(handle: ReturnType<typeof db>, hosts: Map<string, HostAvailability>): Snapshot {
const counts = handle.prepare(`SELECT
COALESCE(SUM(state IN ('running','awaiting-health')),0) AS active,
COALESCE(SUM(state='queued'),0) AS queued FROM server_deployments
WHERE action IN ('create','rebuild') AND state IN ('queued','running','awaiting-health')
AND server_deleted_at IS NULL`).get() as { active: number; queued: number };
return { hosts, ...counts, startupLimit: fleetSettings().createConcurrency, updatedAt: new Date().toISOString() };
}
function observeHost(host: Host): HostAvailability {
const handle = db();
const { running, startup } = capacityCalibration(host.id);
const measured = new Map(host.inventory.instanceResources?.map(sample => [sample.instanceId, sample.generation]));
const pending = handle.prepare(`SELECT DISTINCT i.id,i.generation FROM native_instances i
JOIN native_provisioning n ON n.instance_id=i.id JOIN server_deployments d ON d.id=n.deployment_id
WHERE i.host_id=? AND i.retired_at IS NULL AND d.action IN ('create','rebuild')
AND d.state IN ('running','awaiting-health')`).all(host.id) as { id: string; generation: number }[];
const unmeasured = pending.filter(instance => measured.get(instance.id) !== instance.generation).length;
const { cpuCount, cpuUsagePercent, memoryTotal, memoryAvailable } = host.inventory;
const freeCpu = cpuCount && cpuUsagePercent !== undefined
? Math.max(0, cpuCount * (0.95 - cpuUsagePercent / 100) - unmeasured * startup.cpuCores) : null;
const freeMemory = memoryTotal && memoryAvailable !== undefined && memoryAvailable <= memoryTotal
? Math.max(0, memoryAvailable - 500_000_000 - unmeasured * startup.memoryBytes) : null;
return { host, freeCpu, freeMemory, runningCpu: running.cpuCores, runningMemory: running.memoryBytes,
ports: handle.prepare('SELECT port,bind_address FROM ec_host_ports WHERE host_id=?').all(host.id) as HostAvailability['ports'] };
}
/** Host telemetry updates the estimate once, independently of how many lobbies are previewed. */
export function refreshHostAvailability(host: Host): void {
const handle = db();
if (!snapshotSchemaReady(handle)) return;
const hosts = new Map(snapshots.get(handle)?.hosts);
hosts.set(host.id, observeHost(host));
snapshots.set(handle, queueSnapshot(handle, hosts));
}
/** Lifecycle events refresh the shared snapshot after their transaction and request have yielded. */
export function scheduleAvailabilityRefresh(): void {
const handle = db();
if (scheduled.has(handle)) return;
scheduled.add(handle);
setImmediate(() => {
scheduled.delete(handle);
if (!handle.open) return;
try {
if (!snapshotSchemaReady(handle)) return;
const hosts = new Map(listHosts().map(host => [host.id, observeHost(host)]));
snapshots.set(handle, queueSnapshot(handle, hosts));
} catch (error) { console.error('[availability] snapshot refresh failed:', error); }
});
}
subscribe(event => {
if (event.type === 'fleet-invalidated' || event.type === 'server-status') scheduleAvailabilityRefresh();
});
function freePorts(snapshot: HostAvailability, bindAddress?: string): number {
const bind = bindAddress ?? snapshot.host.bindAddress;
const used = new Set(snapshot.ports.filter(row => row.port >= snapshot.host.portMin && row.port <= snapshot.host.portMax
&& (row.bind_address === bind || ['0.0.0.0', '::'].includes(bind) || ['0.0.0.0', '::'].includes(row.bind_address))).map(row => row.port));
return Math.max(0, snapshot.host.portMax - snapshot.host.portMin + 1 - used.size);
}
/** Estimates only: no placement, historical scans, reservations, or live host requests. */
export function launchAvailability(selections: NativePlacement[]): { availability: LaunchAvailability; warnings: string[] } {
const snapshot = snapshots.get(db());
if (!snapshot) {
scheduleAvailabilityRefresh();
return { availability: { updatedAt: null, requested: selections.length, estimatedAvailable: null, portsAvailable: null,
cpuCoresAvailable: null, memoryBytesAvailable: null, startupSlots: null, startupLimit: null, queued: null },
warnings: selections.length ? ['Capacity measurements are updating. You can queue this launch; provisioning waits for an eligible host.'] : [] };
}
const candidates = [...snapshot.hosts.values()].filter(({ host }) => host.readiness === 'ready'
&& host.lastHeartbeat && Date.now() - Date.parse(host.lastHeartbeat) <= HEARTBEAT_MAX_AGE_MS
&& (host.inventory.storageAvailable ?? 0) >= host.minimumFreeBytes + 1024 ** 3);
const eligible = (candidate: HostAvailability, selection: NativePlacement) => (!selection.hostId || selection.automatic || candidate.host.id === selection.hostId)
&& !!candidate.host.inventory.runtimeImages?.includes(selection.image ?? '')
&& !!candidate.host.inventory.libraries?.some(library => library.id === selection.libraryId && ['ready', 'outdated'].includes(library.state));
const relevant = candidates.filter(candidate => selections.some(selection => eligible(candidate, selection)));
const planned = new Map<string, number>();
let estimatedAvailable = 0;
for (const selection of selections) {
const candidate = relevant.find(candidate => {
const used = planned.get(candidate.host.id) ?? 0;
return eligible(candidate, selection) && freePorts(candidate, selection.network?.bindAddress ?? selection.bindAddress) > used
&& candidate.freeCpu !== null && candidate.freeMemory !== null
&& candidate.freeCpu >= (used + 1) * candidate.runningCpu
&& candidate.freeMemory >= (used + 1) * candidate.runningMemory;
});
if (candidate) { estimatedAvailable++; planned.set(candidate.host.id, (planned.get(candidate.host.id) ?? 0) + 1); }
}
const warnings: string[] = [];
const unknown = relevant.some(candidate => candidate.freeCpu === null || candidate.freeMemory === null);
if (selections.length && !relevant.length) warnings.push('Waiting for an eligible host with the selected runtime and game files. The launch can be queued now.');
else if (unknown) warnings.push('Some host CPU or memory measurements are unavailable. Provisioning waits for fresh capacity measurements.');
else if (estimatedAvailable < selections.length) warnings.push(`Current capacity is estimated for ${estimatedAvailable} of ${selections.length} servers. Remaining launches wait for capacity, including any removals in this collection.`);
const startupSlots = Math.max(0, snapshot.startupLimit - snapshot.active);
if (selections.length && !startupSlots) warnings.push('All startup slots are in use. New launches will wait in the queue.');
if (selections.length && snapshot.queued) warnings.push(`${snapshot.queued} server${snapshot.queued === 1 ? ' is' : 's are'} already queued.`);
return { availability: { updatedAt: snapshot.updatedAt, requested: selections.length, estimatedAvailable: unknown ? null : estimatedAvailable,
portsAvailable: relevant.reduce((total, candidate) => total + freePorts(candidate), 0),
cpuCoresAvailable: unknown ? null : relevant.reduce((total, candidate) => total + (candidate.freeCpu ?? 0), 0),
memoryBytesAvailable: unknown ? null : relevant.reduce((total, candidate) => total + (candidate.freeMemory ?? 0), 0),
startupSlots, startupLimit: snapshot.startupLimit, queued: snapshot.queued }, warnings };
}
+24 -2
View File
@@ -5,6 +5,7 @@ import { isIP } from 'node:net';
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import { ensureHostSchema } from './schema.js'; import { ensureHostSchema } from './schema.js';
import { recordInstanceResources, validateInstanceResources } from './calibration.js'; import { recordInstanceResources, validateInstanceResources } from './calibration.js';
import { refreshHostAvailability, scheduleAvailabilityRefresh } from './availability.js';
import type { Host, HostDecommission, HostDeletion, HostInventory, HostMode, HostOperation, HostOperationState } from '../../hosts.js'; import type { Host, HostDecommission, HostDeletion, HostInventory, HostMode, HostOperation, HostOperationState } from '../../hosts.js';
export const AGENT_PROTOCOL_VERSION = 1; export const AGENT_PROTOCOL_VERSION = 1;
@@ -83,6 +84,17 @@ export function hostDeletionStatus(id: string): HostDeletion {
const names = instances.slice(0, 5).map(instance => instance.server).join(', ') + (instances.length > 5 ? ', …' : ''); const names = instances.slice(0, 5).map(instance => instance.server).join(', ') + (instances.length > 5 ? ', …' : '');
blockers.push(`Remove the ${instances.length} server instance${instances.length === 1 ? '' : 's'} assigned to this host first (${names}). Queued and failed deployments also require removal.`); blockers.push(`Remove the ${instances.length} server instance${instances.length === 1 ? '' : 's'} assigned to this host first (${names}). Queued and failed deployments also require removal.`);
} }
const pinned = handle.prepare(`SELECT d.server FROM server_deployments d
JOIN native_creation_intents q ON q.deployment_id=d.id
LEFT JOIN native_provisioning n ON n.deployment_id=d.id
WHERE d.action='create' AND d.server_deleted_at IS NULL
AND d.state IN ('queued','waiting','running','awaiting-health','interrupted') AND n.deployment_id IS NULL
AND json_extract(q.selection,'$.hostId')=? AND COALESCE(json_extract(q.selection,'$.automatic'),0)!=1
ORDER BY d.server LIMIT 6`).all(id) as { server: string }[];
if (pinned.length) {
const names = pinned.slice(0, 5).map(job => job.server).join(', ') + (pinned.length > 5 ? ', …' : '');
blockers.push(`Cancel the queued server launches pinned to this host before deleting it (${names}).`);
}
const ports = handle.prepare('SELECT COUNT(*) AS count FROM ec_host_ports WHERE host_id=?').get(id) as { count: number }; const ports = handle.prepare('SELECT COUNT(*) AS count FROM ec_host_ports WHERE host_id=?').get(id) as { count: number };
if (ports.count && !instances.length) blockers.push(`This host still has ${ports.count} reserved game port${ports.count === 1 ? '' : 's'}. Finish or reconcile the corresponding server removals first.`); if (ports.count && !instances.length) blockers.push(`This host still has ${ports.count} reserved game port${ports.count === 1 ? '' : 's'}. Finish or reconcile the corresponding server removals first.`);
// A pending command may have reached the agent before its acknowledgement. // A pending command may have reached the agent before its acknowledgement.
@@ -174,6 +186,7 @@ export function createHost(input: Record<string, unknown>): Host {
const bind = input.bindAddress === undefined ? '0.0.0.0' : textField(input.bindAddress, 'Bind address'); const bind = input.bindAddress === undefined ? '0.0.0.0' : textField(input.bindAddress, 'Bind address');
if (!isIP(bind)) throw new Error('Bind address must be an IPv4 or IPv6 address.'); if (!isIP(bind)) throw new Error('Bind address must be an IPv4 or IPv6 address.');
database().prepare('INSERT INTO ec_hosts(id,name,port_min,port_max,bind_address,created_at) VALUES(?,?,?,?,?,?)').run(id, name, min, max, bind, now()); database().prepare('INSERT INTO ec_hosts(id,name,port_min,port_max,bind_address,created_at) VALUES(?,?,?,?,?,?)').run(id, name, min, max, bind, now());
scheduleAvailabilityRefresh();
return requireHost(id); return requireHost(id);
} }
export function updateHost(id: string, input: Record<string, unknown>): Host { export function updateHost(id: string, input: Record<string, unknown>): Host {
@@ -190,6 +203,7 @@ export function updateHost(id: string, input: Record<string, unknown>): Host {
const minimum = input.minimumFreeBytes ?? host.minimumFreeBytes; const minimum = input.minimumFreeBytes ?? host.minimumFreeBytes;
if (typeof minimum !== 'number' || !Number.isSafeInteger(minimum) || minimum < 0) throw new Error('Minimum free storage must be a nonnegative byte count.'); if (typeof minimum !== 'number' || !Number.isSafeInteger(minimum) || minimum < 0) throw new Error('Minimum free storage must be a nonnegative byte count.');
database().prepare('UPDATE ec_hosts SET name=?,mode=?,port_min=?,port_max=?,bind_address=?,public_address=?,minimum_free_bytes=? WHERE id=?').run(name, mode, min, max, bind, publicAddress, minimum, id); database().prepare('UPDATE ec_hosts SET name=?,mode=?,port_min=?,port_max=?,bind_address=?,public_address=?,minimum_free_bytes=? WHERE id=?').run(name, mode, min, max, bind, publicAddress, minimum, id);
scheduleAvailabilityRefresh();
return requireHost(id); return requireHost(id);
} }
@@ -256,6 +270,7 @@ export function enrollHost(token: unknown, rawInventory: unknown): { hostId: str
const credential = issueCredential(enrollment.host_id); const credential = issueCredential(enrollment.host_id);
database().prepare('UPDATE ec_hosts SET enrolled_at=?,last_heartbeat=?,hostname=?,inventory=? WHERE id=?').run(now(), now(), inventory.hostname ?? '', JSON.stringify(inventory), enrollment.host_id); database().prepare('UPDATE ec_hosts SET enrolled_at=?,last_heartbeat=?,hostname=?,inventory=? WHERE id=?').run(now(), now(), inventory.hostname ?? '', JSON.stringify(inventory), enrollment.host_id);
recordResources(enrollment.host_id, inventory, now()); recordResources(enrollment.host_id, inventory, now());
scheduleAvailabilityRefresh();
return { hostId: enrollment.host_id, credential, heartbeatSeconds: 15 }; return { hostId: enrollment.host_id, credential, heartbeatSeconds: 15 };
})(); })();
} }
@@ -270,6 +285,7 @@ export function revokeCredentials(hostId: string): void {
const host = requireHost(hostId); const host = requireHost(hostId);
if (host.decommission && !host.decommission.retryable) throw new Error('Host cleanup needs its credentials until removal is confirmed.'); if (host.decommission && !host.decommission.retryable) throw new Error('Host cleanup needs its credentials until removal is confirmed.');
database().prepare('UPDATE ec_host_credentials SET revoked_at=? WHERE host_id=? AND revoked_at IS NULL').run(now(), hostId); database().prepare('UPDATE ec_host_credentials SET revoked_at=? WHERE host_id=? AND revoked_at IS NULL').run(now(), hostId);
scheduleAvailabilityRefresh();
} }
export function rotateCredential(hostId: string, nextCredential?: string): string { export function rotateCredential(hostId: string, nextCredential?: string): string {
requireHost(hostId); requireHost(hostId);
@@ -292,7 +308,9 @@ export function heartbeat(hostId: string, value: unknown): Host {
database().prepare('UPDATE ec_hosts SET last_heartbeat=?,hostname=?,inventory=? WHERE id=?').run(timestamp, inventory.hostname ?? '', JSON.stringify(inventory), hostId); database().prepare('UPDATE ec_hosts SET last_heartbeat=?,hostname=?,inventory=? WHERE id=?').run(timestamp, inventory.hostname ?? '', JSON.stringify(inventory), hostId);
recordResources(hostId, inventory, timestamp); recordResources(hostId, inventory, timestamp);
})(); })();
return requireHost(hostId); const host = requireHost(hostId);
refreshHostAvailability(host);
return host;
} }
function recordResources(hostId: string, inventory: HostInventory, timestamp: string): void { function recordResources(hostId: string, inventory: HostInventory, timestamp: string): void {
@@ -504,7 +522,11 @@ export function reserveHostPort(hostId: string, instanceId: string, options: { p
const allocated = options.port ?? nextHostPort(hostId, [], bind, protocol); const allocated = options.port ?? nextHostPort(hostId, [], bind, protocol);
if (allocated === null || !hostPortAvailable(hostId, allocated, bind, protocol, instanceId)) throw new Error('No available game port on the selected host.'); if (allocated === null || !hostPortAvailable(hostId, allocated, bind, protocol, instanceId)) throw new Error('No available game port on the selected host.');
database().prepare('INSERT INTO ec_host_ports(host_id,instance_id,bind_address,protocol,port,created_at) VALUES(?,?,?,?,?,?)').run(hostId, instanceId, bind, protocol, allocated, now()); database().prepare('INSERT INTO ec_host_ports(host_id,instance_id,bind_address,protocol,port,created_at) VALUES(?,?,?,?,?,?)').run(hostId, instanceId, bind, protocol, allocated, now());
scheduleAvailabilityRefresh();
return allocated; return allocated;
})(); })();
} }
export function releaseHostPorts(instanceId: string): void { database().prepare('DELETE FROM ec_host_ports WHERE instance_id=?').run(instanceId); } export function releaseHostPorts(instanceId: string): void {
database().prepare('DELETE FROM ec_host_ports WHERE instance_id=?').run(instanceId);
scheduleAvailabilityRefresh();
}
+48 -9
View File
@@ -2,12 +2,46 @@ import { db } from '../db/index.js';
import { config } from '../config.js'; import { config } from '../config.js';
import { AGENT_PROTOCOL_VERSION, getHost, queueHostOperation } from './index.js'; import { AGENT_PROTOCOL_VERSION, getHost, queueHostOperation } from './index.js';
import { getHostLibraryCache, latestPublishedLibraries, queueLibraryDownload } from '../game-files/index.js'; import { getHostLibraryCache, latestPublishedLibraries, queueLibraryDownload } from '../game-files/index.js';
import { GAME_FILE_GAMES } from '../../game-files.js';
import type { Host, HostOperationState, HostPreparation, HostPreparationItem } from '../../hosts.js'; import type { Host, HostOperationState, HostPreparation, HostPreparationItem } from '../../hosts.js';
const activeStates = ['pending', 'accepted', 'running', 'unknown']; const activeStates = ['pending', 'accepted', 'running', 'unknown'];
const supersededDownload = 'Superseded by newer game files before delivery to the host.'; const supersededDownload = 'Superseded by newer game files before delivery to the host.';
type PreparationOperation = { id: string; state: HostOperationState; updated_at: string; error: string | null }; type PreparationOperation = { id: string; state: HostOperationState; updated_at: string; error: string | null };
type Target = { item: HostPreparationItem; image?: string; versionId?: string; requiredBytes: number; operation?: PreparationOperation }; type Target = { item: HostPreparationItem; image?: string; versionId?: string; requiredBytes: number; operation?: PreparationOperation };
type QueuedSelection = { libraryId: string; image: string | null };
type LibraryTarget = Pick<ReturnType<typeof latestPublishedLibraries>[number], 'id' | 'sizeBytes' | 'manifestSha256'> & { name: string; queued?: boolean };
/** Accepted requests retain their prerequisites until placement owns an instance. */
function queuedSelections(hostId: string): QueuedSelection[] {
// Standalone host/library readers can initialize before provisioning migrations.
if (!db().prepare("SELECT 1 FROM sqlite_master WHERE type='table' AND name='native_creation_intents'").get()) return [];
return db().prepare(`SELECT DISTINCT json_extract(q.selection,'$.libraryId') AS libraryId,
json_extract(q.selection,'$.image') AS image FROM native_creation_intents q
JOIN server_deployments d ON d.id=q.deployment_id
LEFT JOIN native_provisioning n ON n.deployment_id=d.id
WHERE d.action='create' AND d.server_deleted_at IS NULL AND n.deployment_id IS NULL
AND d.state IN ('queued','waiting','running','awaiting-health','interrupted')
AND (json_extract(q.selection,'$.automatic')=1 OR json_extract(q.selection,'$.hostId') IS NULL
OR json_extract(q.selection,'$.hostId')=?)`).all(hostId) as QueuedSelection[];
}
function targetLibraries(selections: QueuedSelection[]): LibraryTarget[] {
const latest = latestPublishedLibraries();
const latestIds = new Set(latest.map(version => version.id));
const queued = new Map<string, LibraryTarget>();
const get = db().prepare(`SELECT v.id,l.game,v.version,v.size_bytes AS sizeBytes,v.manifest_sha256 AS manifestSha256
FROM game_file_versions v JOIN game_file_libraries l ON l.id=v.library_id
WHERE v.id=? AND v.state='ready' AND NOT EXISTS(SELECT 1 FROM game_file_cleanup c WHERE c.version_id=v.id)`);
for (const selection of selections) {
if (latestIds.has(selection.libraryId) || queued.has(selection.libraryId)) continue;
const version = get.get(selection.libraryId) as { id: string; game: string; version: string; sizeBytes: number; manifestSha256: string } | undefined;
if (version) queued.set(version.id, { ...version, queued: true,
name: `${GAME_FILE_GAMES.find(game => game.id === version.game)?.name ?? version.game} · Queued version ${version.version}` });
}
// Finish prerequisites for accepted work before downloading newer optional files.
return [...queued.values(), ...latest];
}
function latestOperation(hostId: string, kind: string, field: string, value: string): PreparationOperation | undefined { function latestOperation(hostId: string, kind: string, field: string, value: string): PreparationOperation | undefined {
return db().prepare(`SELECT id,state,updated_at,error FROM ec_host_operations return db().prepare(`SELECT id,state,updated_at,error FROM ec_host_operations
@@ -43,19 +77,24 @@ function hostBlocker(host: Host): { state: 'waiting' | 'blocked'; detail: string
} }
function targets(host: Host): Target[] { function targets(host: Host): Target[] {
const image = config.komodo.defaultImage; const selections = queuedSelections(host.id);
const runtimeReady = host.inventory.runtimeImages?.includes(image); const images = new Set([config.komodo.defaultImage, ...selections.flatMap(selection => selection.image ? [selection.image] : [])]);
const result: Target[] = [{ image, requiredBytes: host.minimumFreeBytes, const result: Target[] = [...images].map(image => {
operation: latestOperation(host.id, 'runtime.pull', 'image', image), const runtimeReady = host.inventory.runtimeImages?.includes(image);
item: { kind: 'runtime', label: image, state: runtimeReady ? 'ready' : 'waiting', detail: runtimeReady ? 'Runtime is cached.' : 'Runtime will be downloaded automatically.' } }]; return { image, requiredBytes: host.minimumFreeBytes,
for (const version of latestPublishedLibraries()) { operation: latestOperation(host.id, 'runtime.pull', 'image', image),
item: { kind: 'runtime', label: image, state: runtimeReady ? 'ready' : 'waiting', detail: runtimeReady ? 'Runtime is cached.' : 'Runtime will be downloaded automatically.' } };
});
for (const version of targetLibraries(selections)) {
const cache = getHostLibraryCache(host.id, version.id); const cache = getHostLibraryCache(host.id, version.id);
const matches = cache?.manifestSha256 === version.manifestSha256; const matches = cache?.manifestSha256 === version.manifestSha256;
const ready = matches && ['ready', 'outdated'].includes(cache!.status) && cache!.bytesDownloaded === version.sizeBytes; const ready = matches && ['ready', 'outdated'].includes(cache!.status) && cache!.bytesDownloaded === version.sizeBytes;
result.push({ versionId: version.id, requiredBytes: version.sizeBytes - (matches ? cache!.bytesDownloaded : 0) + host.minimumFreeBytes, result.push({ versionId: version.id, requiredBytes: version.sizeBytes - (matches ? cache!.bytesDownloaded : 0) + host.minimumFreeBytes,
operation: latestOperation(host.id, 'library.download', 'libraryVersionId', version.id), operation: latestOperation(host.id, 'library.download', 'libraryVersionId', version.id),
item: { kind: 'library', label: version.name, state: ready ? 'ready' : 'waiting', item: { kind: 'library', label: version.name, state: ready ? 'ready' : 'waiting',
detail: ready ? 'The latest game files are downloaded and verified.' : 'The latest game files will be downloaded and verified automatically.', detail: version.queued
? ready ? 'Game files selected by queued launches are downloaded and verified.' : 'Game files selected by queued launches will be downloaded and verified automatically.'
: ready ? 'The latest game files are downloaded and verified.' : 'The latest game files will be downloaded and verified automatically.',
bytesDownloaded: cache?.bytesDownloaded ?? 0, bytesTotal: version.sizeBytes, ...(cache ? { cache } : {}) } }); bytesDownloaded: cache?.bytesDownloaded ?? 0, bytesTotal: version.sizeBytes, ...(cache ? { cache } : {}) } });
} }
const blocker = hostBlocker(host); const blocker = hostBlocker(host);
@@ -100,13 +139,13 @@ export function presentHostPreparation(host: Host): Host {
} }
function supersedeUndeliveredDownloads(hostId: string): void { function supersedeUndeliveredDownloads(hostId: string): void {
const latest = new Set(latestPublishedLibraries().map(version => version.id)); const required = new Set(targetLibraries(queuedSelections(hostId)).map(version => version.id));
const pending = db().prepare(`SELECT id,json_extract(payload,'$.libraryVersionId') AS versionId const pending = db().prepare(`SELECT id,json_extract(payload,'$.libraryVersionId') AS versionId
FROM ec_host_operations WHERE host_id=? AND kind='library.download' AND state='pending' FROM ec_host_operations WHERE host_id=? AND kind='library.download' AND state='pending'
AND last_delivered_at IS NULL AND json_extract(payload,'$.automatic')=1`).all(hostId) as { id: string; versionId: string }[]; AND last_delivered_at IS NULL AND json_extract(payload,'$.automatic')=1`).all(hostId) as { id: string; versionId: string }[];
const now = new Date().toISOString(); const now = new Date().toISOString();
for (const operation of pending) { for (const operation of pending) {
if (latest.has(operation.versionId)) continue; if (required.has(operation.versionId)) continue;
// Delivered operations may already be writing files. Only unsubmitted // Delivered operations may already be writing files. Only unsubmitted
// automatic work can be replaced without consulting the agent journal. // automatic work can be replaced without consulting the agent journal.
db().prepare("UPDATE ec_host_operations SET state='failed',error=?,updated_at=? WHERE id=?").run(supersededDownload, now, operation.id); db().prepare("UPDATE ec_host_operations SET state='failed',error=?,updated_at=? WHERE id=?").run(supersededDownload, now, operation.id);
+2
View File
@@ -3,6 +3,8 @@ import type Database from 'better-sqlite3';
const initialized = new WeakSet<Database.Database>(); const initialized = new WeakSet<Database.Database>();
export function ensureHostSchema(handle: Database.Database): void { export function ensureHostSchema(handle: Database.Database): void {
if (initialized.has(handle)) return; if (initialized.has(handle)) return;
// Background readers open only after controller migrations have completed.
if (handle.readonly) { initialized.add(handle); return; }
handle.exec(` handle.exec(`
CREATE TABLE IF NOT EXISTS ec_hosts ( CREATE TABLE IF NOT EXISTS ec_hosts (
id TEXT PRIMARY KEY, name TEXT NOT NULL, hostname TEXT NOT NULL DEFAULT '', id TEXT PRIMARY KEY, name TEXT NOT NULL, hostname TEXT NOT NULL DEFAULT '',
+15 -8
View File
@@ -6,9 +6,13 @@ import { recordProgress } from './progress.js';
export function canCancelCreation(id: string): boolean { export function canCancelCreation(id: string): boolean {
return !!db().prepare(`SELECT 1 FROM server_deployments d return !!db().prepare(`SELECT 1 FROM server_deployments d
JOIN native_provisioning n ON n.deployment_id=d.id LEFT JOIN native_provisioning n ON n.deployment_id=d.id
JOIN native_instances i ON i.id=n.instance_id AND i.generation=n.generation AND i.retired_at IS NULL LEFT JOIN native_instances i ON i.id=n.instance_id AND i.generation=n.generation AND i.retired_at IS NULL
WHERE d.id=? AND d.action='create' AND d.server_deleted_at IS NULL WHERE d.id=? AND d.action='create' AND d.server_deleted_at IS NULL
AND (i.id IS NOT NULL OR (n.deployment_id IS NULL
AND EXISTS (SELECT 1 FROM native_creation_intents intent WHERE intent.deployment_id=d.id)
AND NOT EXISTS (SELECT 1 FROM native_instances replacement WHERE replacement.server=d.server AND replacement.retired_at IS NULL)
AND NOT EXISTS (SELECT 1 FROM servers replacement WHERE replacement.name=d.server AND replacement.destroyed_at IS NULL)))
AND d.state IN ('queued','waiting','running','awaiting-health','interrupted','failed') AND d.state IN ('queued','waiting','running','awaiting-health','interrupted','failed')
AND NOT EXISTS (SELECT 1 FROM server_deployments other WHERE other.server=d.server AND NOT EXISTS (SELECT 1 FROM server_deployments other WHERE other.server=d.server
AND other.server_deleted_at IS NULL AND other.rowid>d.rowid AND other.state!='cancelled' AND other.server_deleted_at IS NULL AND other.rowid>d.rowid AND other.state!='cancelled'
@@ -23,9 +27,9 @@ export function cancelCreation(id: string, triggeredBy = 'ui'): string | null {
if (prior) return prior.cleanup_id; if (prior) return prior.cleanup_id;
if (!canCancelCreation(id)) return null; if (!canCancelCreation(id)) return null;
const job = db().prepare(`SELECT d.server,n.operation_id,n.stage_operation_id FROM server_deployments d const job = db().prepare(`SELECT d.server,n.operation_id,n.stage_operation_id FROM server_deployments d
JOIN native_provisioning n ON n.deployment_id=d.id WHERE d.id=?`).get(id) as LEFT JOIN native_provisioning n ON n.deployment_id=d.id WHERE d.id=?`).get(id) as
{ server: string; operation_id: string | null; stage_operation_id: string | null }; { server: string; operation_id: string | null; stage_operation_id: string | null };
const instance = nativeInstance(job.server)!; const instance = nativeInstance(job.server);
// Replace a removal still waiting on occupancy, including one whose asynchronous probe is outstanding. // Replace a removal still waiting on occupancy, including one whose asynchronous probe is outstanding.
for (const removal of db().prepare("SELECT id FROM server_deployments WHERE server=? AND action='destroy' AND state='waiting' AND server_deleted_at IS NULL").all(job.server) as { id: string }[]) { for (const removal of db().prepare("SELECT id FROM server_deployments WHERE server=? AND action='destroy' AND state='waiting' AND server_deleted_at IS NULL").all(job.server) as { id: string }[]) {
cancelRemoval(removal.id); cancelRemoval(removal.id);
@@ -37,13 +41,16 @@ export function cancelCreation(id: string, triggeredBy = 'ui'): string | null {
event: { kind: 'cancelled', message: 'Creation cancelled. Its partial instance will be removed without a player-count check.', details: { cleanupId } } }); event: { kind: 'cancelled', message: 'Creation cancelled. Its partial instance will be removed without a player-count check.', details: { cleanupId } } });
if (!job.operation_id && !job.stage_operation_id) { if (!job.operation_id && !job.stage_operation_id) {
// No remote mutation was ever queued. The reservation is the only resource to remove. // No remote mutation was ever queued. The reservation is the only resource to remove.
db().prepare('UPDATE native_instances SET retired_at=? WHERE id=?').run(new Date().toISOString(), instance.id); if (instance) {
releaseHostPorts(instance.id); db().prepare('UPDATE native_instances SET retired_at=? WHERE id=?').run(new Date().toISOString(), instance.id);
releaseHostPorts(instance.id);
}
deploymentStage(cleanupId, 'Unsubmitted creation cancelled; reservations released', { phase: 'cleanup', deploymentStage(cleanupId, 'Unsubmitted creation cancelled; reservations released', { phase: 'cleanup',
checkpoint: { key: 'cleanup', label: 'No host command submitted; local reservations released' } }); checkpoint: { key: 'cleanup', label: 'No host command submitted; local reservations released' } });
finishDeployment(cleanupId, { ok: true }); finishDeployment(cleanupId, { ok: true });
finishCreationCancellation(cleanupId); finishCreationCancellation(cleanupId);
} else { } else {
if (!instance) throw new Error('The original native instance is unavailable.');
const operation = queueHostOperation(instance.host_id, 'instance.cancel-create', { createOperationId: job.operation_id ?? id }, const operation = queueHostOperation(instance.host_id, 'instance.cancel-create', { createOperationId: job.operation_id ?? id },
{ instanceId: instance.id, generation: instance.generation, idempotencyKey: cleanupId }); { instanceId: instance.id, generation: instance.generation, idempotencyKey: cleanupId });
db().prepare('UPDATE native_provisioning SET operation_id=? WHERE deployment_id=?').run(operation.id, cleanupId); db().prepare('UPDATE native_provisioning SET operation_id=? WHERE deployment_id=?').run(operation.id, cleanupId);
@@ -57,12 +64,12 @@ export function cancelCreation(id: string, triggeredBy = 'ui'): string | null {
export function finishCreationCancellation(cleanupId: string): void { export function finishCreationCancellation(cleanupId: string): void {
const cancellation = db().prepare(`SELECT c.creation_id,n.operation_id FROM native_creation_cancellations c const cancellation = db().prepare(`SELECT c.creation_id,n.operation_id FROM native_creation_cancellations c
JOIN native_provisioning n ON n.deployment_id=c.creation_id WHERE c.cleanup_id=?`).get(cleanupId) as LEFT JOIN native_provisioning n ON n.deployment_id=c.creation_id WHERE c.cleanup_id=?`).get(cleanupId) as
{ creation_id: string; operation_id: string | null } | undefined; { creation_id: string; operation_id: string | null } | undefined;
if (!cancellation) return; if (!cancellation) return;
const now = new Date().toISOString(); const now = new Date().toISOString();
db().prepare('UPDATE server_deployments SET server_deleted_at=? WHERE id IN (?,?)').run(now, cancellation.creation_id, cleanupId); db().prepare('UPDATE server_deployments SET server_deleted_at=? WHERE id IN (?,?)').run(now, cancellation.creation_id, cleanupId);
deploymentStage(cancellation.creation_id, 'Creation cancelled; partial instance removed', { blocker: null, deploymentStage(cancellation.creation_id, 'Creation cancelled; cleanup completed', { blocker: null,
checkpoint: { key: 'cleanup', phase: 'cleanup', label: 'Cancellation cleanup confirmed; instance and reservations removed' } }); checkpoint: { key: 'cleanup', phase: 'cleanup', label: 'Cancellation cleanup confirmed; instance and reservations removed' } });
// The original command may have lost its receipt. Cleanup proves that its effects are now gone. // The original command may have lost its receipt. Cleanup proves that its effects are now gone.
if (cancellation.operation_id) { if (cancellation.operation_id) {
+5 -3
View File
@@ -9,6 +9,7 @@ import { readProgress, recordProgress, type ProgressUpdate } from './progress.js
import { bindNativeOperation } from './native.js'; import { bindNativeOperation } from './native.js';
import { workerActivity } from '../pipeline/activity.js'; import { workerActivity } from '../pipeline/activity.js';
import { canCancelCreation } from './cancel-creation.js'; import { canCancelCreation } from './cancel-creation.js';
import { nativeDeployment } from '../backends/native.js';
export function beginDeployment(server: string, action: ProvisioningAction, triggeredBy = 'ui'): string { export function beginDeployment(server: string, action: ProvisioningAction, triggeredBy = 'ui'): string {
const id = randomUUID(); const id = randomUUID();
@@ -103,7 +104,7 @@ export function getDeployment(id: string, options: { events?: 'all' | 'recent' }
(ProvisioningDeployment & { observation_json: string | null; phase_key: string | null; phase_started_at: string | null; observed_at: string | null }) | undefined; (ProvisioningDeployment & { observation_json: string | null; phase_key: string | null; phase_started_at: string | null; observed_at: string | null }) | undefined;
if (!row) return null; if (!row) return null;
const generatedAt = new Date().toISOString(); const generatedAt = new Date().toISOString();
const native = db().prepare('SELECT instance_id FROM native_provisioning WHERE deployment_id=?').get(id); const native = nativeDeployment(id);
const agentOperation = native ? db().prepare(`SELECT o.id, o.host_id AS hostId, o.state, o.updated_at AS updatedAt, o.created_at AS createdAt, o.error const agentOperation = native ? db().prepare(`SELECT o.id, o.host_id AS hostId, o.state, o.updated_at AS updatedAt, o.created_at AS createdAt, o.error
FROM native_provisioning n JOIN ec_host_operations o ON o.id=n.operation_id WHERE n.deployment_id=?`).get(id) as ProvisioningDetail['agentOperation'] : null; FROM native_provisioning n JOIN ec_host_operations o ON o.id=n.operation_id WHERE n.deployment_id=?`).get(id) as ProvisioningDetail['agentOperation'] : null;
const cancellation = db().prepare(`SELECT c.cleanup_id AS cleanupId,d.state,d.stage,d.error FROM native_creation_cancellations c const cancellation = db().prepare(`SELECT c.cleanup_id AS cleanupId,d.state,d.stage,d.error FROM native_creation_cancellations c
@@ -254,10 +255,11 @@ export function listDeployments() {
export function interruptDeployments(): void { export function interruptDeployments(): void {
db().transaction(() => { db().transaction(() => {
const running = db().prepare("SELECT id FROM server_deployments WHERE state = 'running' AND id NOT IN (SELECT deployment_id FROM native_provisioning)").all() as { id: string }[]; const native = 'SELECT deployment_id FROM native_provisioning UNION ALL SELECT deployment_id FROM native_creation_intents';
const running = db().prepare(`SELECT id FROM server_deployments WHERE state = 'running' AND id NOT IN (${native})`).all() as { id: string }[];
const finishedAt = new Date().toISOString(); const finishedAt = new Date().toISOString();
const error = 'The application restarted before this operation finished. Check the server before retrying.'; const error = 'The application restarted before this operation finished. Check the server before retrying.';
db().prepare(`UPDATE server_deployments SET state = 'interrupted', finished_at = ?, error = ? WHERE state = 'running' AND id NOT IN (SELECT deployment_id FROM native_provisioning)`).run(finishedAt, error); db().prepare(`UPDATE server_deployments SET state = 'interrupted', finished_at = ?, error = ? WHERE state = 'running' AND id NOT IN (${native})`).run(finishedAt, error);
for (const { id } of running) recordProgress(id, { blocker: null, event: { kind: 'outcome', for (const { id } of running) recordProgress(id, { blocker: null, event: { kind: 'outcome',
message: 'Interrupted by application restart. Check the server before retrying.', details: { state: 'interrupted', finishedAt, error } } }); message: 'Interrupted by application restart. Check the server before retrying.', details: { state: 'interrupted', finishedAt, error } } });
})(); })();
+1 -1
View File
@@ -44,7 +44,7 @@ export type CreateServerFailure =
| 'db'; // shouldn't happen, but wraps INSERT failures | 'db'; // shouldn't happen, but wraps INSERT failures
export type CreateServerResult = export type CreateServerResult =
| { ok: true; name: string; port: number; deploymentId: string } | { ok: true; name: string; port: number | null; deploymentId: string }
| { ok: false; kind: CreateServerFailure; message: string }; | { ok: false; kind: CreateServerFailure; message: string };
export type DestroyServerResult = export type DestroyServerResult =
@@ -4,6 +4,7 @@ import { config } from '../config.js';
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import type { NativeInstance } from '../backends/native.js'; import type { NativeInstance } from '../backends/native.js';
import type { LaunchRecipePlan } from '../fleet/launch-recipes.js'; import type { LaunchRecipePlan } from '../fleet/launch-recipes.js';
import type { PlacementOptions } from '../hosts/capacity.js';
export interface NativeLaunchInput { export interface NativeLaunchInput {
game: 't5' | 't6'; game: 't5' | 't6';
@@ -25,7 +26,7 @@ export interface PreparedNativeLaunch {
let worker: Worker | null = null; let worker: Worker | null = null;
let sequence = 0; let sequence = 0;
const pending = new Map<number, { const pending = new Map<number, {
resolve: (value: PreparedNativeLaunch) => void; resolve: (value: unknown) => void;
reject: (error: Error) => void; reject: (error: Error) => void;
timer: NodeJS.Timeout; timer: NodeJS.Timeout;
}>(); }>();
@@ -39,14 +40,14 @@ function stop(error: Error, current = worker): Promise<number> | undefined {
} }
/** The reader and JSON encoder run off the controller's request thread. */ /** The reader and JSON encoder run off the controller's request thread. */
export function prepareNativeLaunch(input: NativeLaunchInput): Promise<PreparedNativeLaunch> { function run<T>(kind: 'launch' | 'placement', input: NativeLaunchInput | PlacementOptions): Promise<T> {
db(); // Migrations must finish before the independent read-only connection opens. db(); // Migrations must finish before the independent read-only connection opens.
if (!worker) { if (!worker) {
const current = worker = new Worker(resolve(process.env.NODE_ENV === 'production' const current = worker = new Worker(resolve(process.env.NODE_ENV === 'production'
? 'build/launch/worker.mjs' : '.svelte-kit/launch/worker.mjs'), { ? 'build/launch/worker.mjs' : '.svelte-kit/launch/worker.mjs'), {
workerData: { dbPath: resolve(config.dbPath) } workerData: { dbPath: resolve(config.dbPath) }
}); });
current.on('message', ({ id, value, error }: { id: number; value: PreparedNativeLaunch; error?: string }) => { current.on('message', ({ id, value, error }: { id: number; value: unknown; error?: string }) => {
if (current !== worker) return; if (current !== worker) return;
const request = pending.get(id); const request = pending.get(id);
if (!request) return; if (!request) return;
@@ -59,13 +60,18 @@ export function prepareNativeLaunch(input: NativeLaunchInput): Promise<PreparedN
} }
worker.ref(); worker.ref();
const id = ++sequence; const id = ++sequence;
return new Promise((resolve, reject) => { return new Promise<T>((resolve, reject) => {
const timer = setTimeout(() => { void stop(new Error('Launch preparation timed out; preparation will retry.')); }, 60_000); const timer = setTimeout(() => { void stop(new Error('Launch preparation timed out; preparation will retry.')); }, 60_000);
pending.set(id, { resolve, reject, timer }); pending.set(id, { resolve: value => resolve(value as T), reject, timer });
worker!.postMessage({ id, input }); worker!.postMessage({ id, kind, input });
}); });
} }
export function prepareNativeLaunch(input: NativeLaunchInput): Promise<PreparedNativeLaunch> { return run('launch', input); }
export function evaluateNativePlacement(input: PlacementOptions): Promise<{ hostId: string; lastHeartbeat: string | null }> {
return run('placement', input);
}
export async function closeLaunchPreparation(): Promise<void> { export async function closeLaunchPreparation(): Promise<void> {
await stop(new Error('Launch preparation stopped.')); await stop(new Error('Launch preparation stopped.'));
} }
+9 -1
View File
@@ -5,9 +5,17 @@ import { relDestForKind } from '../config.js';
import { captureRecipe, summarizeRecipe } from '../fleet/launch-recipes.js'; import { captureRecipe, summarizeRecipe } from '../fleet/launch-recipes.js';
import { sha256Buffer } from '../fleet/files.js'; import { sha256Buffer } from '../fleet/files.js';
import type { NativeLaunchInput } from './launch-preparation.js'; import type { NativeLaunchInput } from './launch-preparation.js';
import { choosePlacement, type PlacementOptions } from '../hosts/capacity.js';
parentPort!.on('message', ({ id, input }: { id: number; input: NativeLaunchInput }) => { parentPort!.on('message', (message: { id: number; kind: 'launch'; input: NativeLaunchInput } | { id: number; kind: 'placement'; input: PlacementOptions }) => {
const { id } = message;
try { try {
if (message.kind === 'placement') {
const decision = db().transaction(() => choosePlacement(message.input))();
parentPort!.postMessage({ id, value: { hostId: decision.host.id, lastHeartbeat: decision.host.lastHeartbeat } });
return;
}
const input = message.input;
// All file versions and composed configuration belong to one snapshot. // All file versions and composed configuration belong to one snapshot.
const recipe = db().transaction(() => ({ ...captureRecipe(input.game, input.mapSlug), const recipe = db().transaction(() => ({ ...captureRecipe(input.game, input.mapSlug),
native: input.native, image: input.instance.image }))(); native: input.native, image: input.instance.image }))();
+113 -57
View File
@@ -1,15 +1,15 @@
import { isIP } from 'node:net'; import { isIP } from 'node:net';
import { setImmediate as yieldToController } from 'node:timers/promises'; import { setImmediate as yieldToController } from 'node:timers/promises';
import { CapacityError, choosePlacement, availableHostPort, type PlacementOptions } from '../hosts/capacity.js'; import { choosePlacement, availableHostPort, type PlacementOptions } from '../hosts/capacity.js';
import { randomBytes, randomUUID } from 'node:crypto'; import { randomBytes, randomUUID } from 'node:crypto';
import { posix } from 'node:path'; import { posix } from 'node:path';
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import { config, relDestForKind, serverDir } from '../config.js'; import { config, relDestForKind, serverDir } from '../config.js';
import { nativeInstance, nativeRequest, type NativeInstance } from '../backends/native.js'; import { nativeInstance, nativeRequest, unplacedNativeCreation, type NativeInstance } from '../backends/native.js';
import { getHost, getHostOperationStatus, listHosts, queueHostOperation, queueHostOperationSerialized, releaseHostPorts, reserveHostPort } from '../hosts/index.js'; import { getHost, getHostOperationStatus, listHosts, queueHostOperation, queueHostOperationSerialized, releaseHostPorts, reserveHostPort } from '../hosts/index.js';
import { newestReadyLibraryId, assertReadyLibrary } from '../game-files/index.js'; import { newestReadyLibraryId, assertReadyLibrary } from '../game-files/index.js';
import { blueprintRecipe, captureRecipePlan, recipeForDeployment, recipeStorage, summarizeRecipe, type LaunchRecipePlan } from '../fleet/launch-recipes.js'; import { blueprintRecipe, captureRecipePlan, recipeForDeployment, recipeStorage, summarizeRecipe, type LaunchRecipePlan } from '../fleet/launch-recipes.js';
import { prepareNativeLaunch, type PreparedNativeLaunch } from './launch-preparation.js'; import { prepareNativeLaunch, evaluateNativePlacement, type PreparedNativeLaunch } from './launch-preparation.js';
import { getServerKey, saveServerKey } from '../fleet/server-keys.js'; import { getServerKey, saveServerKey } from '../fleet/server-keys.js';
import { activeBetaRemovalBlocker, getServer } from '../fleet/servers.js'; import { activeBetaRemovalBlocker, getServer } from '../fleet/servers.js';
import { nextLobbyNumber, type CreateServerInput, type CreateServerResult } from './instances.js'; import { nextLobbyNumber, type CreateServerInput, type CreateServerResult } from './instances.js';
@@ -30,11 +30,12 @@ import { cancelCreation, finishCreationCancellation } from './cancel-creation.js
export type NativeSelection = import('../../hosts.js').NativePlacement; export type NativeSelection = import('../../hosts.js').NativePlacement;
type NativeRecipe = LaunchRecipePlan & { native?: NativeSelection }; type NativeRecipe = LaunchRecipePlan & { native?: NativeSelection };
interface Job { id: string; server: string; action: ProvisioningAction; state: string; submitted_at: string | null; stage: string; interface Job { selection?: string | null; id: string; server: string; action: ProvisioningAction; state: string; submitted_at: string | null; stage: string;
instance_id: string; generation: number; stage_operation_id: string | null; operation_id: string | null; files_json: string | null; applied_at: string | null } instance_id: string; generation: number; stage_operation_id: string | null; operation_id: string | null; files_json: string | null; applied_at: string | null }
interface FilePlan { id: number; path: string; contentBase64: string | null; sha256: string | null; checkBefore: boolean; expectedBeforeSha: string | null } interface FilePlan { id: number; path: string; contentBase64: string | null; sha256: string | null; checkBefore: boolean; expectedBeforeSha: string | null }
export function nativeSelection(input: Record<string, unknown>, game: string, options: Pick<PlacementOptions, 'planned'> = {}): NativeSelection { /** Validate a launch request without selecting a host or consuming capacity. */
export function nativeLaunchSelection(input: Record<string, unknown>, game: string): NativeSelection {
if (input.hostId !== undefined && typeof input.hostId !== 'string') throw new Error('Invalid host.'); if (input.hostId !== undefined && typeof input.hostId !== 'string') throw new Error('Invalid host.');
const libraryId = newestReadyLibraryId(game); const libraryId = newestReadyLibraryId(game);
if (!libraryId) throw new Error(`Upload ${game.toUpperCase()} game files before launching a server.`); if (!libraryId) throw new Error(`Upload ${game.toUpperCase()} game files before launching a server.`);
@@ -44,16 +45,29 @@ export function nativeSelection(input: Record<string, unknown>, game: string, op
const image = input.image === undefined || input.image === '' ? config.komodo.defaultImage : input.image; const image = input.image === undefined || input.image === '' ? config.komodo.defaultImage : input.image;
if (typeof image !== 'string' || !/^[a-zA-Z0-9][a-zA-Z0-9._/:@-]{0,499}$/.test(image)) throw new Error('Invalid runtime image.'); if (typeof image !== 'string' || !/^[a-zA-Z0-9][a-zA-Z0-9._/:@-]{0,499}$/.test(image)) throw new Error('Invalid runtime image.');
const automatic = input.automatic === true || !input.hostId; const automatic = input.automatic === true || !input.hostId;
if (!automatic) {
const host = getHost(input.hostId as string);
if (!host) throw new Error('The selected host was not found.');
if (host.decommission) throw new Error('The selected host is being removed. Choose another host.');
}
const rawNetwork = input.automatic === true ? input.network ?? {} : input; const rawNetwork = input.automatic === true ? input.network ?? {} : input;
if (!rawNetwork || typeof rawNetwork !== 'object' || Array.isArray(rawNetwork)) throw new Error('Invalid network overrides.'); if (!rawNetwork || typeof rawNetwork !== 'object' || Array.isArray(rawNetwork)) throw new Error('Invalid network overrides.');
const network = rawNetwork as NativeSelection; const network = rawNetwork as NativeSelection;
const bindAddress = network.bindAddress || undefined; const bindAddress = network.bindAddress || undefined;
if (bindAddress !== undefined && (typeof bindAddress !== 'string' || !isIP(bindAddress))) throw new Error('Invalid bind address.'); if (bindAddress !== undefined && (typeof bindAddress !== 'string' || !isIP(bindAddress))) throw new Error('Invalid bind address.');
const { host } = choosePlacement({ hostId: automatic ? undefined : input.hostId as string, libraryId, image, bindAddress, ...options });
if (network.publicAddress !== undefined && (typeof network.publicAddress !== 'string' || network.publicAddress.length > 253)) throw new Error('Invalid public address.'); if (network.publicAddress !== undefined && (typeof network.publicAddress !== 'string' || network.publicAddress.length > 253)) throw new Error('Invalid public address.');
if (network.publicPort !== undefined && (!Number.isInteger(network.publicPort) || Number(network.publicPort) < 1 || Number(network.publicPort) > 65535)) throw new Error('Invalid public port.'); if (network.publicPort !== undefined && (!Number.isInteger(network.publicPort) || Number(network.publicPort) < 1 || Number(network.publicPort) > 65535)) throw new Error('Invalid public port.');
return { hostId: host.id, libraryId, image, bindAddress: bindAddress ?? host.bindAddress, publicAddress: network.publicAddress ?? host.publicAddress, return { hostId: automatic ? undefined : input.hostId as string, libraryId, image, automatic,
publicPort: network.publicPort, automatic, network: { bindAddress, publicAddress: network.publicAddress, publicPort: network.publicPort } }; bindAddress, publicAddress: network.publicAddress, publicPort: network.publicPort,
network: { bindAddress, publicAddress: network.publicAddress, publicPort: network.publicPort } };
}
export function nativeSelection(input: Record<string, unknown>, game: string, options: Pick<PlacementOptions, 'planned'> = {}): NativeSelection {
const selection = nativeLaunchSelection(input, game);
const { host } = choosePlacement({ hostId: selection.hostId, libraryId: selection.libraryId,
image: selection.image!, bindAddress: selection.bindAddress, ...options });
return { ...selection, hostId: host.id, bindAddress: selection.bindAddress ?? host.bindAddress,
publicAddress: selection.publicAddress ?? host.publicAddress };
} }
export function previewNativeCreation(input: Record<string, unknown>): LaunchPreview { export function previewNativeCreation(input: Record<string, unknown>): LaunchPreview {
@@ -79,7 +93,7 @@ export function previewNativeCreation(input: Record<string, unknown>): LaunchPre
db().prepare('INSERT INTO server_launch_previews(id,expires_at,server,port,recipe,blueprint_name) VALUES(?,?,?,?,?,?)') db().prepare('INSERT INTO server_launch_previews(id,expires_at,server,port,recipe,blueprint_name) VALUES(?,?,?,?,?,?)')
.run(id, expiresAt, name, port, JSON.stringify(recipe), blueprintName); .run(id, expiresAt, name, port, JSON.stringify(recipe), blueprintName);
const checks: LaunchPreview['checks'] = [ const checks: LaunchPreview['checks'] = [
{ label: 'Host capacity', state: 'pass', detail: selection.automatic ? 'Capacity available. The least loaded eligible host is selected when provisioning starts.' : `${getHost(selection.hostId)!.name}: capacity available.` }, { label: 'Host capacity', state: 'pass', detail: selection.automatic ? 'Capacity available. The least loaded eligible host is selected when provisioning starts.' : `${getHost(selection.hostId!)!.name}: capacity available.` },
{ label: 'Game files', state: 'pass', detail: `The latest ${recipe.game.toUpperCase()} game files are verified on an eligible host.` }, { label: 'Game files', state: 'pass', detail: `The latest ${recipe.game.toUpperCase()} game files are verified on an eligible host.` },
{ label: 'Lobby', state: name ? 'pass' : 'blocked', detail: name ?? 'All lobby IDs for this map are reserved.' }, { label: 'Lobby', state: name ? 'pass' : 'blocked', detail: name ?? 'All lobby IDs for this map are reserved.' },
{ label: 'Host port', state: port ? 'pass' : 'blocked', detail: port ? selection.automatic ? 'UDP/TCP capacity available; port assigned at placement.' : `${selection.bindAddress}:${port} / UDP and TCP` : 'No available UDP/TCP port on this host.' }, { label: 'Host port', state: port ? 'pass' : 'blocked', detail: port ? selection.automatic ? 'UDP/TCP capacity available; port assigned at placement.' : `${selection.bindAddress}:${port} / UDP and TCP` : 'No available UDP/TCP port on this host.' },
@@ -90,7 +104,7 @@ export function previewNativeCreation(input: Record<string, unknown>): LaunchPre
} }
export function nextNativePort(selection: NativeSelection, taken: number[] = []): number | null { export function nextNativePort(selection: NativeSelection, taken: number[] = []): number | null {
const host = getHost(selection.hostId); const host = selection.hostId ? getHost(selection.hostId) : null;
return host ? availableHostPort(host, { bindAddress: selection.bindAddress, return host ? availableHostPort(host, { bindAddress: selection.bindAddress,
planned: taken.map(port => ({ hostId: host.id, port })) }) : null; planned: taken.map(port => ({ hostId: host.id, port })) }) : null;
} }
@@ -99,42 +113,51 @@ export function queueNativeCreation(input: CreateServerInput): CreateServerResul
try { try {
return db().transaction(() => { return db().transaction(() => {
const preview = input.previewId ? db().prepare('SELECT * FROM server_launch_previews WHERE id=?').get(input.previewId) as const preview = input.previewId ? db().prepare('SELECT * FROM server_launch_previews WHERE id=?').get(input.previewId) as
{ id: string; recipe: string; server: string; port: number; expires_at: string; deployment_id: string | null; blueprint_name: string | null } | undefined : undefined; { id: string; recipe: string; server: string; port: number | null; expires_at: string; deployment_id: string | null; blueprint_name: string | null } | undefined : undefined;
if (input.previewId && !preview) throw new Error('Review a fresh launch preview.'); if (input.previewId && !preview) throw new Error('Review a fresh launch preview.');
if (preview?.deployment_id) return { ok: true as const, name: preview.server, port: preview.port, deploymentId: preview.deployment_id }; if (preview?.deployment_id) return { ok: true as const, name: preview.server, port: preview.port, deploymentId: preview.deployment_id };
if (preview && preview.expires_at <= new Date().toISOString()) throw new Error('Launch preview expired.'); if (preview && preview.expires_at <= new Date().toISOString()) throw new Error('Launch preview expired.');
const saved = preview ? JSON.parse(preview.recipe) as NativeRecipe : undefined; const saved = preview ? JSON.parse(preview.recipe) as NativeRecipe : undefined;
const game = input.game ?? 't5'; const game = input.game ?? 't5';
if (input.gameMode !== 'zombie' || (saved && (saved.game !== game || saved.mapSlug !== input.mapSlug))) throw new Error('Selection changed. Review a fresh launch preview.'); if (input.gameMode !== 'zombie' || (saved && (saved.game !== game || saved.mapSlug !== input.mapSlug))) throw new Error('Selection changed. Review a fresh launch preview.');
const selection = nativeSelection((saved?.native ?? input) as unknown as Record<string, unknown>, game); const selection = nativeLaunchSelection((saved?.native ?? input) as unknown as Record<string, unknown>, game);
const recipe: NativeRecipe = { ...captureRecipePlan(game, input.mapSlug), native: selection, image: selection.image! };
const lobby = nextLobbyNumber(`${game}_zombie_${input.mapSlug}`); const lobby = nextLobbyNumber(`${game}_zombie_${input.mapSlug}`);
if (lobby === null) throw new Error('All lobby IDs are reserved.'); if (lobby === null) throw new Error('All lobby IDs are reserved.');
const name = `${game}_zombie_${input.mapSlug}_${lobby}`; const name = `${game}_zombie_${input.mapSlug}_${lobby}`;
const port = nextNativePort(selection); if ((input.expectedName && input.expectedName !== name) || (preview && preview.server !== name)) throw new Error('Available lobby changed. Review a fresh preview.');
if (!port) throw new Error('No available host port.'); const result = queueNativeCreationIntent({ server: name, game, mapSlug: input.mapSlug, selection,
if ((input.expectedName && input.expectedName !== name) || (preview && (preview.server !== name || (!selection.automatic && preview.port !== port)))) throw new Error('Available lobby or host port changed. Review a fresh preview.'); plutoKey: input.plutoKey, createdBy: input.createdBy, blueprintName: preview?.blueprint_name,
const key = input.plutoKey?.trim() || getServerKey(name); recipe: saved ?? captureRecipePlan(game, input.mapSlug) });
if (!key) throw new Error(`Save a Plutonium server key for ${name}.`); if (result.ok && preview) db().prepare('UPDATE server_launch_previews SET deployment_id=?,port=NULL WHERE id=?').run(result.deploymentId, preview.id);
saveServerKey(name, key); return result;
const instanceId = randomUUID(); })();
reserveHostPort(selection.hostId, instanceId, { port, bindAddress: selection.bindAddress, protocol: 'udp' }); } catch (error) { return { ok: false, kind: 'bad-request', message: (error as Error).message }; }
reserveHostPort(selection.hostId, instanceId, { port, bindAddress: selection.bindAddress, protocol: 'tcp' }); }
db().prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,bind_address,port,public_address,public_port,created_at)
VALUES(?,?,?,?,?,?,?,?,?,?)`).run(instanceId, name, selection.hostId, selection.libraryId, selection.image, selection.bindAddress, port, /** Persist the reviewed logical identity; all host/port/resource decisions belong to provisioning. */
selection.publicAddress ?? null, selection.publicPort ?? null, new Date().toISOString()); export function queueNativeCreationIntent(input: { server: string; game: 't5' | 't6'; mapSlug: string; selection: NativeSelection;
const id = beginDeployment(name, 'create', input.createdBy); plutoKey?: string; createdBy?: string; blueprintName?: string | null; recipe?: LaunchRecipePlan }): CreateServerResult {
try {
return db().transaction(() => {
const expected = `${input.game}_zombie_${input.mapSlug}_${nextLobbyNumber(`${input.game}_zombie_${input.mapSlug}`)}`;
if (input.server !== expected || nativeInstance(input.server)) throw new Error('Available lobby changed. Review a fresh preview.');
const key = input.plutoKey?.trim() || getServerKey(input.server);
if (!key) throw new Error(`Save a Plutonium server key for ${input.server}.`);
saveServerKey(input.server, key);
const recipe: NativeRecipe = { ...(input.recipe ?? captureRecipePlan(input.game, input.mapSlug)),
native: input.selection, image: input.selection.image! };
const id = beginDeployment(input.server, 'create', input.createdBy ?? 'ui');
db().prepare("UPDATE server_deployments SET state='queued' WHERE id=?").run(id); db().prepare("UPDATE server_deployments SET state='queued' WHERE id=?").run(id);
db().prepare('INSERT INTO server_creation_jobs(deployment_id,server,port,game_mode,map_slug,expected_map) VALUES(?,?,?,?,?,?)') db().prepare('INSERT INTO server_creation_jobs(deployment_id,server,port,game_mode,map_slug,expected_map) VALUES(?,?,NULL,?,?,?)')
.run(id, name, port, 'zombie', recipe.mapSlug, recipe.expectedMap); .run(id, input.server, 'zombie', input.mapSlug, recipe.expectedMap);
db().prepare('INSERT INTO native_creation_intents(deployment_id,selection) VALUES(?,?)').run(id, JSON.stringify(input.selection));
const stored = recipeStorage(recipe); const stored = recipeStorage(recipe);
db().prepare('INSERT INTO server_launch_recipes(deployment_id,recipe,recipe_summary,blueprint_name) VALUES(?,?,?,?)') db().prepare('INSERT INTO server_launch_recipes(deployment_id,recipe,recipe_summary,blueprint_name) VALUES(?,?,?,?)')
.run(id, stored.recipe, stored.recipeSummary, preview?.blueprint_name ?? null); .run(id, stored.recipe, stored.recipeSummary, input.blueprintName ?? null);
if (preview) db().prepare('UPDATE server_launch_previews SET deployment_id=?,port=? WHERE id=?').run(id, port, preview.id); deploymentStage(id, 'Awaiting placement', { phase: 'eligibility', recipeStatus: 'queued' });
deploymentStage(id, 'Waiting for provisioning evaluation', { phase: 'eligibility', recipeStatus: 'queued' }); return { ok: true as const, name: input.server, port: null, deploymentId: id };
return { ok: true as const, name, port, deploymentId: id };
})(); })();
} catch (error) { return { ok: false, kind: error instanceof CapacityError ? 'conflict' : 'bad-request', message: (error as Error).message }; } } catch (error) { return { ok: false, kind: 'bad-request', message: (error as Error).message }; }
} }
export function bindNativeOperation(id: string, server: string): void { export function bindNativeOperation(id: string, server: string): void {
@@ -144,8 +167,11 @@ export function bindNativeOperation(id: string, server: string): void {
/** Cancelling an unsubmitted launch releases only its local reservation. */ /** Cancelling an unsubmitted launch releases only its local reservation. */
export function cancelQueuedNativeCreation(server: string, triggeredBy: string): string | null { export function cancelQueuedNativeCreation(server: string, triggeredBy: string): string | null {
if (getServer(server)) return null;
const unplaced = unplacedNativeCreation(server);
if (unplaced) return cancelCreation(unplaced, triggeredBy);
const instance = nativeInstance(server); const instance = nativeInstance(server);
if (!instance || getServer(server)) return null; if (!instance) return null;
return db().transaction(() => { return db().transaction(() => {
const creation = db().prepare(`SELECT d.id FROM server_deployments d JOIN native_provisioning n ON n.deployment_id=d.id const creation = db().prepare(`SELECT d.id FROM server_deployments d JOIN native_provisioning n ON n.deployment_id=d.id
WHERE n.instance_id=? AND d.action='create' AND d.state='queued' AND n.operation_id IS NULL`).get(instance.id) as { id: string } | undefined; WHERE n.instance_id=? AND d.action='create' AND d.state='queued' AND n.operation_id IS NULL`).get(instance.id) as { id: string } | undefined;
@@ -243,9 +269,10 @@ export async function processNativeProvisioning(eligibility: (operation: { id: s
deploymentStage(id, 'Host draining; waiting for a fresh empty lobby'); deploymentStage(id, 'Host draining; waiting for a fresh empty lobby');
} }
} }
const jobs = db().prepare(`SELECT d.*,n.instance_id,n.generation,n.stage_operation_id,n.operation_id,n.files_json,n.applied_at const jobs = db().prepare(`SELECT d.*,n.instance_id,n.generation,n.stage_operation_id,n.operation_id,n.files_json,n.applied_at,q.selection
FROM native_provisioning n JOIN server_deployments d ON d.id=n.deployment_id FROM server_deployments d LEFT JOIN native_provisioning n ON n.deployment_id=d.id
WHERE d.server_deleted_at IS NULL AND d.state IN ('queued','waiting','running','awaiting-health','interrupted') ORDER BY d.started_at,d.rowid`).all() as Job[]; LEFT JOIN native_creation_intents q ON q.deployment_id=d.id
WHERE (n.deployment_id IS NOT NULL OR q.deployment_id IS NOT NULL) AND d.server_deleted_at IS NULL AND d.state IN ('queued','waiting','running','awaiting-health','interrupted') ORDER BY d.started_at,d.rowid`).all() as Job[];
// One host's observation must not prevent another job being admitted. // One host's observation must not prevent another job being admitted.
// Per-job ownership also covers overlapping direct calls and worker passes. // Per-job ownership also covers overlapping direct calls and worker passes.
const work = jobs.map(job => { const work = jobs.map(job => {
@@ -254,7 +281,7 @@ export async function processNativeProvisioning(eligibility: (operation: { id: s
const work = trackActivity(`provisioning:${job.id}`, `Evaluating ${job.action} for ${job.server}`, const work = trackActivity(`provisioning:${job.id}`, `Evaluating ${job.action} for ${job.server}`,
() => processJob(job, eligibility)) () => processJob(job, eligibility))
.catch(error => { if (stillActive(job)) hold(job, (error as Error).message); }) .catch(error => { if (stillActive(job)) hold(job, (error as Error).message); })
.finally(() => { activeJobs.delete(job.id); }); .finally(() => { activeJobs.delete(job.id); preparingJobs.delete(job.id); });
activeJobs.set(job.id, work); activeJobs.set(job.id, work);
return work; return work;
}); });
@@ -281,27 +308,47 @@ function sameLaunchFiles(first: LaunchRecipePlan, second: LaunchRecipePlan): boo
return JSON.stringify(selected(first)) === JSON.stringify(selected(second)); return JSON.stringify(selected(first)) === JSON.stringify(selected(second));
} }
/** Reservations can move only while no remote work has ever been submitted. */ /** Capacity history and host ranking run in the background reader, one job at a time. */
function placeQueuedCreation(job: Job, instance: NativeInstance): NativeInstance { async function placeQueuedCreation(job: Job, instance: NativeInstance | null): Promise<NativeInstance | null> {
const recipe = recipeForDeployment(job.id)!.recipe as NativeRecipe;
const prior: NativeSelection = job.selection ? JSON.parse(job.selection) : recipe.native!;
const network = prior.automatic ? prior.network ?? {} : prior;
const options: PlacementOptions = { hostId: prior.automatic ? undefined : prior.hostId,
libraryId: prior.libraryId, image: prior.image ?? recipe.image, bindAddress: network.bindAddress,
thorough: true, dispatch: true, excludeInstanceId: instance?.id };
const decision = await evaluateNativePlacement(options);
return db().transaction(() => { return db().transaction(() => {
const recipe = recipeForDeployment(job.id)!.recipe as NativeRecipe; const state = db().prepare('SELECT state FROM server_deployments WHERE id=? AND server_deleted_at IS NULL').get(job.id) as { state: string } | undefined;
const prior = recipe.native!; if (state?.state !== 'queued') return null;
const network = prior.automatic ? prior.network ?? {} : prior; const live = nativeInstance(job.server);
const { host, port } = choosePlacement({ hostId: prior.automatic ? undefined : instance.host_id, if (instance ? live?.id !== instance.id || live.generation !== instance.generation : !!live || !!getServer(job.server)) return null;
libraryId: instance.library_id, image: instance.image, bindAddress: network.bindAddress, const active = (db().prepare(`SELECT count(*) AS n FROM server_deployments WHERE id!=? AND server_deleted_at IS NULL
thorough: true, dispatch: true, excludeInstanceId: instance.id }); AND action IN ('create','rebuild') AND state IN ('running','awaiting-health')`).get(job.id) as { n: number }).n;
if (active + otherPreparations(job.id) >= fleetSettings().createConcurrency) { hold(job, 'Waiting for startup capacity.'); return null; }
if (getHost(decision.hostId)?.lastHeartbeat !== decision.lastHeartbeat) { hold(job, 'Host measurements changed; waiting for another placement check.'); return null; }
// A short current-state check protects against reservations made while the
// reader ranked hosts. It never scans resource history or contacts agents.
const { host, port } = choosePlacement({ ...options, hostId: decision.hostId, thorough: false });
const bind = network.bindAddress ?? host.bindAddress; const bind = network.bindAddress ?? host.bindAddress;
releaseHostPorts(instance.id); const instanceId = instance?.id ?? randomUUID();
reserveHostPort(host.id, instance.id, { port, bindAddress: bind, protocol: 'udp' }); if (instance) releaseHostPorts(instanceId);
reserveHostPort(host.id, instance.id, { port, bindAddress: bind, protocol: 'tcp' }); reserveHostPort(host.id, instanceId, { port, bindAddress: bind, protocol: 'udp' });
db().prepare(`UPDATE native_instances SET host_id=?,bind_address=?,port=?,public_address=?,public_port=? WHERE id=?`) reserveHostPort(host.id, instanceId, { port, bindAddress: bind, protocol: 'tcp' });
.run(host.id, bind, port, network.publicAddress ?? host.publicAddress, network.publicPort ?? null, instance.id); if (instance) db().prepare(`UPDATE native_instances SET host_id=?,bind_address=?,port=?,public_address=?,public_port=? WHERE id=?`)
.run(host.id, bind, port, network.publicAddress ?? host.publicAddress, network.publicPort ?? null, instanceId);
else {
db().prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,bind_address,port,public_address,public_port,created_at)
VALUES(?,?,?,?,?,?,?,?,?,?)`).run(instanceId, job.server, host.id, prior.libraryId, options.image, bind, port,
network.publicAddress ?? host.publicAddress, network.publicPort ?? null, new Date().toISOString());
db().prepare('INSERT INTO native_provisioning(deployment_id,instance_id,generation) VALUES(?,?,1)').run(job.id, instanceId);
}
db().prepare('UPDATE server_creation_jobs SET port=? WHERE deployment_id=?').run(port, job.id); db().prepare('UPDATE server_creation_jobs SET port=? WHERE deployment_id=?').run(port, job.id);
recipe.native = { ...prior, hostId: host.id, bindAddress: bind, publicAddress: network.publicAddress ?? host.publicAddress, publicPort: network.publicPort }; recipe.native = { ...prior, hostId: host.id, bindAddress: bind, publicAddress: network.publicAddress ?? host.publicAddress, publicPort: network.publicPort };
const stored = recipeStorage(recipe); const stored = recipeStorage(recipe);
db().prepare('UPDATE server_launch_recipes SET recipe=?,recipe_summary=? WHERE deployment_id=?').run(stored.recipe, stored.recipeSummary, job.id); db().prepare('UPDATE server_launch_recipes SET recipe=?,recipe_summary=? WHERE deployment_id=?').run(stored.recipe, stored.recipeSummary, job.id);
db().prepare('UPDATE server_launch_previews SET port=? WHERE deployment_id=?').run(port, job.id); db().prepare('UPDATE server_launch_previews SET port=? WHERE deployment_id=?').run(port, job.id);
deploymentStage(job.id, `Placed on ${host.name} after CPU, memory, history and port checks`, { phase: 'eligibility' }); deploymentStage(job.id, `Placed on ${host.name} after CPU, memory, history and port checks`, { phase: 'eligibility' });
preparingJobs.add(job.id);
return nativeInstance(job.server)!; return nativeInstance(job.server)!;
})(); })();
} }
@@ -309,15 +356,24 @@ function placeQueuedCreation(job: Job, instance: NativeInstance): NativeInstance
async function processJob(job: Job, eligibility: (job: Job) => string | null) { async function processJob(job: Job, eligibility: (job: Job) => string | null) {
if (!stillActive(job)) return; if (!stillActive(job)) return;
let instance = nativeInstance(job.server); let instance = nativeInstance(job.server);
if (!instance || instance.id !== job.instance_id || (instance.generation !== job.generation && !(job.action === 'rebuild' && instance.generation === job.generation + 1))) {
finishDeployment(job.id, { ok: false, message: 'Original instance was retired or replaced; no command was sent.' }); return;
}
if (job.action === 'create' && !job.operation_id && !job.stage_operation_id && !job.submitted_at) { if (job.action === 'create' && !job.operation_id && !job.stage_operation_id && !job.submitted_at) {
// A resumed bound job owns its original identity. An unplaced intent
// cannot adopt an instance or registered server that appeared later.
if (job.instance_id ? !instance || instance.id !== job.instance_id || instance.generation !== job.generation
: !!instance || !!getServer(job.server)) {
finishDeployment(job.id, { ok: false, message: 'Original instance was retired or replaced; no command was sent.' }); return;
}
const settings = fleetSettings(); const settings = fleetSettings();
const active = db().prepare(`SELECT COUNT(*) AS n FROM server_deployments WHERE id!=? AND server_deleted_at IS NULL const active = db().prepare(`SELECT COUNT(*) AS n FROM server_deployments WHERE id!=? AND server_deleted_at IS NULL
AND action IN ('create','rebuild') AND state IN ('running','awaiting-health')`).get(job.id) as { n: number }; AND action IN ('create','rebuild') AND state IN ('running','awaiting-health')`).get(job.id) as { n: number };
if (active.n + otherPreparations(job.id) >= settings.createConcurrency) { hold(job, 'Waiting for startup capacity.'); return; } if (active.n + otherPreparations(job.id) >= settings.createConcurrency) { hold(job, 'Waiting for startup capacity.'); return; }
instance = placeQueuedCreation(job, instance); if (eligibility(job)) return;
instance = await placeQueuedCreation(job, instance);
if (!instance) return;
job.instance_id = instance.id; job.generation = instance.generation;
}
if (!instance || instance.id !== job.instance_id || (instance.generation !== job.generation && !(job.action === 'rebuild' && instance.generation === job.generation + 1))) {
finishDeployment(job.id, { ok: false, message: 'Original instance was retired or replaced; no command was sent.' }); return;
} }
const host = getHost(instance.host_id); const host = getHost(instance.host_id);
if (!host?.connected || host.mode === 'disabled') { if (!host?.connected || host.mode === 'disabled') {
@@ -448,7 +504,7 @@ async function processJob(job: Job, eligibility: (job: Job) => string | null) {
if (job.action === 'destroy' && count(['destroy']) >= settings.destroyConcurrency) { hold(job, 'Waiting for destruction capacity.'); return; } if (job.action === 'destroy' && count(['destroy']) >= settings.destroyConcurrency) { hold(job, 'Waiting for destruction capacity.'); return; }
deploymentStage(job.id, 'Checking host eligibility', { phase: 'eligibility', blocker: null }); deploymentStage(job.id, 'Checking host eligibility', { phase: 'eligibility', blocker: null });
if (job.action === 'create') choosePlacement({ hostId: instance.host_id, libraryId: instance.library_id, image: instance.image, if (job.action === 'create') choosePlacement({ hostId: instance.host_id, libraryId: instance.library_id, image: instance.image,
bindAddress: instance.bind_address, excludeInstanceId: instance.id, thorough: true, dispatch: true }); bindAddress: instance.bind_address, excludeInstanceId: instance.id, thorough: false, dispatch: true });
if (job.action === 'create' || job.action === 'rebuild') assertReadyLibrary(instance.host_id, instance.library_id); if (job.action === 'create' || job.action === 'rebuild') assertReadyLibrary(instance.host_id, instance.library_id);
const checkOccupancy = async (): Promise<boolean> => { const checkOccupancy = async (): Promise<boolean> => {
// Fast controller check keeps occupied lobbies out of the operation slots. // Fast controller check keeps occupied lobbies out of the operation slots.
@@ -514,7 +570,7 @@ async function processJob(job: Job, eligibility: (job: Job) => string | null) {
} }
assertReadyLibrary(instance!.host_id, instance!.library_id); assertReadyLibrary(instance!.host_id, instance!.library_id);
if (job.action === 'create') choosePlacement({ hostId: instance!.host_id, libraryId: instance!.library_id, image: instance!.image, if (job.action === 'create') choosePlacement({ hostId: instance!.host_id, libraryId: instance!.library_id, image: instance!.image,
bindAddress: instance!.bind_address, excludeInstanceId: instance!.id, thorough: true, dispatch: true }); bindAddress: instance!.bind_address, excludeInstanceId: instance!.id, thorough: false, dispatch: true });
} }
if (job.action === 'start' || job.action === 'restart' || job.action === 'rebuild') { if (job.action === 'start' || job.action === 'restart' || job.action === 'rebuild') {
db().prepare(`INSERT INTO restart_rollout(server,requested_at,operation_id) VALUES(?,?,?) ON CONFLICT(server) db().prepare(`INSERT INTO restart_rollout(server,requested_at,operation_id) VALUES(?,?,?) ON CONFLICT(server)
+2 -2
View File
@@ -7,7 +7,7 @@ import { queueRebuildServer } from './instances.js';
import { restartRequestInFlight } from './restarts.js'; import { restartRequestInFlight } from './restarts.js';
import { runtimeActions, type RuntimeAction } from '../../provisioning.js'; import { runtimeActions, type RuntimeAction } from '../../provisioning.js';
import { recordProgress } from './progress.js'; import { recordProgress } from './progress.js';
import { nativeInstance } from '../backends/native.js'; import { nativeInstance, unplacedNativeCreation } from '../backends/native.js';
export interface LifecycleOperation { export interface LifecycleOperation {
id: string; server: string; action: string; state: string; stage: string; id: string; server: string; action: string; state: string; stage: string;
@@ -137,7 +137,7 @@ export function releaseRestartRequest(server: string, source: RestartSource): vo
export function requestLifecycle(server: string, action: RuntimeAction) { export function requestLifecycle(server: string, action: RuntimeAction) {
if (!runtimeActions.includes(action)) return { ok: false as const, status: 400, error: 'Choose start, restart, rebuild, or stop.' }; if (!runtimeActions.includes(action)) return { ok: false as const, status: 400, error: 'Choose start, restart, rebuild, or stop.' };
if (!nativeInstance(server) && (!komodoConfigured() || !rconConfigured())) return { ok: false as const, status: 503, error: 'Configure a host and RCON before provisioning.' }; if (!nativeInstance(server) && !unplacedNativeCreation(server) && (!komodoConfigured() || !rconConfigured())) return { ok: false as const, status: 503, error: 'Configure a host and RCON before provisioning.' };
return db().transaction(() => { return db().transaction(() => {
const blocker = action === 'rebuild' ? activeBetaRemovalBlocker(server) : null; const blocker = action === 'rebuild' ? activeBetaRemovalBlocker(server) : null;
if (blocker) return { ok: false as const, status: 409, error: blocker }; if (blocker) return { ok: false as const, status: 409, error: blocker };
@@ -31,7 +31,7 @@
const keysReady = $derived(preview?.servers.filter(server => server.name && (server.hasSavedKey || keys[server.name]?.trim())).length ?? 0); const keysReady = $derived(preview?.servers.filter(server => server.name && (server.hasSavedKey || keys[server.name]?.trim())).length ?? 0);
onMount(() => { const timer = setInterval(() => { now = Date.now(); }, 10_000); return () => clearInterval(timer); }); onMount(() => { const timer = setInterval(() => { now = Date.now(); }, 10_000); return () => clearInterval(timer); });
let canLaunch = $derived(preview?.ready && preview.mode === mode && (preview.servers.length > 0 || preview.removals.length > 0) let canLaunch = $derived(preview?.ready && preview.mode === mode && (preview.servers.length > 0 || preview.removals.length > 0)
&& (!preview.servers.length || hostReady) && (!preview.removals.length || confirmRemovals) && preview.servers.every(server => server.name && (server.hasSavedKey || keys[server.name]?.trim()))); && (!preview.removals.length || confirmRemovals) && preview.servers.every(server => server.name && (server.hasSavedKey || keys[server.name]?.trim())));
$effect(() => { $effect(() => {
if (!dialog) return; if (!dialog) return;
@@ -54,7 +54,7 @@
} }
async function review() { async function review() {
if (!hostReady) { error = 'Wait for an eligible host and wait for its latest game files and runtime to finish automatic setup.'; return; } if (!hostReady) { error = 'Choose the latest uploaded game files and a valid runtime image before reviewing the launch.'; return; }
const current = ++session; const current = ++session;
const previous = preview; const previous = preview;
busy = true; error = ''; keyNotice = ''; confirmRemovals = false; busy = true; error = ''; keyNotice = ''; confirmRemovals = false;
@@ -105,7 +105,7 @@
{#if open && games.length}<NativeHostSelection {games} bind:selection={hostSelection} bind:ready={hostReady} disabled={busy} onchange={() => { preview = null; confirmRemovals = false; }} />{/if} {#if open && games.length}<NativeHostSelection {games} bind:selection={hostSelection} bind:ready={hostReady} disabled={busy} onchange={() => { preview = null; confirmRemovals = false; }} />{/if}
<p class="muted">{mode === 'exact' ? 'Keep matching lobbies, add missing ones, and remove excess lobbies and maps outside this collection across both games.' : 'Create missing lobbies and keep all existing servers.'}</p> <p class="muted">{mode === 'exact' ? 'Keep matching lobbies, add missing ones, and remove excess lobbies and maps outside this collection across both games.' : 'Create missing lobbies and keep all existing servers.'}</p>
{#if error}<p class="error" role="alert">{error}</p>{/if} {#if error}<p class="error" role="alert">{error}</p>{/if}
{#if !preview}<p class="muted">{busy ? 'Preparing the collection…' : 'Wait for an eligible host. Its latest game files sync automatically before you review capacity and server keys.'}</p>{#if !busy}<Button disabled={!hostReady} onclick={review}>Preview launch</Button>{/if} {#if !preview}<p class="muted">{busy ? 'Preparing the collection…' : 'Review the fleet changes, capacity estimate, and server keys. Queued launches wait automatically when hosts need more capacity or preparation.'}</p>{#if !busy}<Button disabled={!hostReady} onclick={review}>Preview launch</Button>{/if}
{:else} {:else}
{#if keyNotice}<p role="status" class="muted">{keyNotice}</p>{/if} {#if keyNotice}<p role="status" class="muted">{keyNotice}</p>{/if}
{#if expired}<p class="error" role="status">This preview expired. Refresh before launch; keys remain entered for unchanged IDs.</p>{/if} {#if expired}<p class="error" role="status">This preview expired. Refresh before launch; keys remain entered for unchanged IDs.</p>{/if}
@@ -115,10 +115,24 @@
{#if preview.mode === 'exact' && preview.retainedServers.length}<details class="coverage"><summary>Servers to keep ({preview.retainedServers.length})</summary><ul>{#each preview.retainedServers as name}<li><ServerIdentity {name} /><span>{name}</span></li>{/each}</ul></details>{/if} {#if preview.mode === 'exact' && preview.retainedServers.length}<details class="coverage"><summary>Servers to keep ({preview.retainedServers.length})</summary><ul>{#each preview.retainedServers as name}<li><ServerIdentity {name} /><span>{name}</span></li>{/each}</ul></details>{/if}
{#if preview.mode === 'fill' && preview.entries.some(entry => entry.existing)}<details class="coverage" open={!preview.servers.length}><summary>Review existing servers</summary>{#each preview.entries.filter(entry => entry.existing) as entry}<div class="coverage-map"><strong>{entry.recipe.mapName}: {entry.existing} existing</strong><p class="muted">{[...new Set((entry.coverage ?? []).map(server => server.health))].map(health => `${entry.coverage.filter(server => server.health === health).length} ${health}`).join(', ')}</p><ul>{#each entry.coverage ?? [] as server}<li><a href={server.href}><ServerIdentity name={server.name} /></a><span>{server.health === 'online' ? 'Healthy' : server.health}</span></li>{/each}</ul></div>{/each}</details>{/if} {#if preview.mode === 'fill' && preview.entries.some(entry => entry.existing)}<details class="coverage" open={!preview.servers.length}><summary>Review existing servers</summary>{#each preview.entries.filter(entry => entry.existing) as entry}<div class="coverage-map"><strong>{entry.recipe.mapName}: {entry.existing} existing</strong><p class="muted">{[...new Set((entry.coverage ?? []).map(server => server.health))].map(health => `${entry.coverage.filter(server => server.health === health).length} ${health}`).join(', ')}</p><ul>{#each entry.coverage ?? [] as server}<li><a href={server.href}><ServerIdentity name={server.name} /></a><span>{server.health === 'online' ? 'Healthy' : server.health}</span></li>{/each}</ul></div>{/each}</details>{/if}
</div> </div>
{#if preview.availability && preview.servers.length}
<section class="availability" aria-label="Latest reported capacity">
<h3>Latest reported capacity</h3>
<p class="muted">{preview.availability.estimatedAvailable === null ? 'An estimate is not available yet.' : `${preview.availability.estimatedAvailable} additional servers estimated to fit.`} {preview.availability.requested} requested.</p>
<dl>
<div><dt>CPU available</dt><dd>{preview.availability.cpuCoresAvailable === null ? 'Unknown' : `${preview.availability.cpuCoresAvailable.toFixed(1)} cores`}</dd></div>
<div><dt>Memory available</dt><dd>{preview.availability.memoryBytesAvailable === null ? 'Unknown' : `${(preview.availability.memoryBytesAvailable / 1024 ** 3).toFixed(1)} GiB`}</dd></div>
<div><dt>Ports available</dt><dd>{preview.availability.portsAvailable ?? 'Unknown'}</dd></div>
<div><dt>Startup slots</dt><dd>{preview.availability.startupSlots ?? 'Unknown'}{#if preview.availability.startupLimit !== null} / {preview.availability.startupLimit}{/if}</dd></div>
</dl>
<p class="caption">{preview.availability.updatedAt ? `Updated ${$timeFormat.dateTime(preview.availability.updatedAt)}.` : 'Waiting for host reports.'}{#if preview.availability.queued !== null} {preview.availability.queued} servers already queued.{/if} Provisioning checks capacity separately for each server before it starts.</p>
</section>
{/if}
<div class="section-title"><h3>{preview.mode === 'exact' ? 'Review fleet changes' : preview.servers.length ? 'Your launch order' : 'Nothing to create'}</h3><button class="text-button" disabled={busy} onclick={review}>Refresh preview</button></div> <div class="section-title"><h3>{preview.mode === 'exact' ? 'Review fleet changes' : preview.servers.length ? 'Your launch order' : 'Nothing to create'}</h3><button class="text-button" disabled={busy} onclick={review}>Refresh preview</button></div>
{#each preview.blockers as blocker}<p class="error" role="alert">{blocker}</p>{/each} {#each preview.blockers as blocker}<p class="error" role="alert">{blocker}</p>{/each}
{#each preview.warnings ?? [] as warning}<p class="warning" role="status">{warning}</p>{/each}
{#if preview.servers.length || preview.removals.length} {#if preview.servers.length || preview.removals.length}
{#if preview.servers.length}<p class="muted">Saved keys are reused automatically. Add keys only where required below. Update saved keys in <a href="/server-keys" target="_blank" rel="noopener">Keys (opens in a new tab)</a>. Fleet counts and available slots are checked again when you queue the launch.</p>{/if} {#if preview.servers.length}<p class="muted">Saved keys are reused automatically. Add keys only where required below. Update saved keys in <a href="/server-keys" target="_blank" rel="noopener">Keys (opens in a new tab)</a>. Apply saves the reviewed jobs and opens their progress page. Each launch waits for a startup slot and host capacity.</p>{/if}
<form onsubmit={event => { event.preventDefault(); void launch(); }}> <form onsubmit={event => { event.preventDefault(); void launch(); }}>
{#if preview.removals.length} {#if preview.removals.length}
<section class="removals" aria-label="Servers to remove"> <section class="removals" aria-label="Servers to remove">
@@ -133,7 +147,7 @@
{#each preview.servers as server, index (server.name ?? `unavailable-${index}`)} {#each preview.servers as server, index (server.name ?? `unavailable-${index}`)}
<li class="server-row"> <li class="server-row">
<span class="order" aria-hidden="true">{String(index + 1).padStart(2, '0')}</span> <span class="order" aria-hidden="true">{String(index + 1).padStart(2, '0')}</span>
<div class="server-identity">{#if index === 0 || preview.servers[index - 1].entryIndex !== server.entryIndex}<span class="map-group">{preview.entries[server.entryIndex].recipe.game.toUpperCase()} · {preview.entries[server.entryIndex].recipe.mapName}</span>{/if}<strong>{#if server.name}<ServerIdentity name={server.name} />{:else}{preview.entries[server.entryIndex].recipe.mapName}{/if}</strong><span class="server-id">{server.name ?? 'No available lobby'} · {server.port === null ? 'No available capacity' : server.native?.automatic ? 'Host and port assigned automatically' : `Port ${server.port}`}</span></div> <div class="server-identity">{#if index === 0 || preview.servers[index - 1].entryIndex !== server.entryIndex}<span class="map-group">{preview.entries[server.entryIndex].recipe.game.toUpperCase()} · {preview.entries[server.entryIndex].recipe.mapName}</span>{/if}<strong>{#if server.name}<ServerIdentity name={server.name} />{:else}{preview.entries[server.entryIndex].recipe.mapName}{/if}</strong><span class="server-id">{server.name ?? 'No available lobby'} · {server.native ? 'Host and port assigned when provisioning starts' : server.port === null ? 'No available port' : `Port ${server.port}`}</span></div>
{#if server.name} {#if server.name}
{#if server.hasSavedKey}<p class="key-ready"><span aria-hidden="true">✓</span> Saved key ready</p> {#if server.hasSavedKey}<p class="key-ready"><span aria-hidden="true">✓</span> Saved key ready</p>
{:else}<label><span>Server key required</span><input aria-label={`Key for ${server.name}`} data-missing={!keys[server.name]?.trim()} type="password" bind:value={keys[server.name]} disabled={busy} maxlength="512" autocomplete="off" placeholder="Paste this server’s key" /></label>{/if} {:else}<label><span>Server key required</span><input aria-label={`Key for ${server.name}`} data-missing={!keys[server.name]?.trim()} type="password" bind:value={keys[server.name]} disabled={busy} maxlength="512" autocomplete="off" placeholder="Paste this server’s key" /></label>{/if}
@@ -151,6 +165,10 @@
</dialog> </dialog>
<style> <style>
.availability { display: grid; gap: 12px; padding: 16px; border: 1px solid var(--border-subtle); border-radius: var(--radius-md); }
.availability dl { display: grid; grid-template-columns: repeat(auto-fit, minmax(130px, 1fr)); gap: 16px; margin: 0; }
.availability dt { font-size: var(--text-xs); color: var(--text-secondary); }.availability dd { margin: 6px 0 0; font-size: 14px; font-variant-numeric: tabular-nums; }
.warning { color: var(--state-warning); font-size: 13px; line-height: 1.6; }
select { padding: 10px; border: 1px solid var(--border-control); border-radius: var(--radius-md); background: var(--surface-sunken); color: var(--text-primary); font: inherit; } select { padding: 10px; border: 1px solid var(--border-control); border-radius: var(--radius-md); background: var(--surface-sunken); color: var(--text-primary); font: inherit; }
.removals { display: grid; gap: 12px; padding: 16px; border: 1px solid var(--state-danger); border-radius: var(--radius-md); } .removals { display: grid; gap: 12px; padding: 16px; border: 1px solid var(--state-danger); border-radius: var(--radius-md); }
.removals ul { list-style: none; margin: 0; padding: 0; display: grid; gap: 16px; } .removals ul { list-style: none; margin: 0; padding: 0; display: grid; gap: 16px; }
@@ -15,7 +15,8 @@
const latest = $derived(latestGameFiles(libraries)); const latest = $derived(latestGameFiles(libraries));
function cacheReady(game: ZombieGame) { return !!latest[game]?.caches.some(cache => eligible.some(host => host.id === cache.hostId) && ['ready', 'outdated'].includes(cache.status)); } function cacheReady(game: ZombieGame) { return !!latest[game]?.caches.some(cache => eligible.some(host => host.id === cache.hostId) && ['ready', 'outdated'].includes(cache.status)); }
$effect(() => { ready = !loading && !error && games.length > 0 && games.every(game => selection.libraryIds[game] === latest[game]?.id && cacheReady(game)); }); $effect(() => { ready = !loading && !error && /^[a-zA-Z0-9][a-zA-Z0-9._/:@-]{0,499}$/.test(selection.image)
&& games.length > 0 && games.every(game => !!latest[game]?.id && selection.libraryIds[game] === latest[game]?.id); });
$effect(() => { $effect(() => {
if (disabled || !defaultRuntimeImage || runtimeSelection === selection) return; if (disabled || !defaultRuntimeImage || runtimeSelection === selection) return;
if (!selection.image) selection.image = defaultRuntimeImage; if (!selection.image) selection.image = defaultRuntimeImage;
@@ -42,14 +43,14 @@
<div class="title"><h3>Host and game files</h3><button type="button" disabled={disabled || loading} onclick={() => { onchange?.(); void load(); }}>Refresh hosts</button></div> <div class="title"><h3>Host and game files</h3><button type="button" disabled={disabled || loading} onclick={() => { onchange?.(); void load(); }}>Refresh hosts</button></div>
{#if error}<p role="alert" class="error">{error}</p>{/if} {#if error}<p role="alert" class="error">{error}</p>{/if}
{#if loading}<p role="status">Loading host readiness and game files…</p>{/if} {#if loading}<p role="status">Loading host readiness and game files…</p>{/if}
<p>Hosts are assigned automatically. We check available capacity before accepting a launch and choose the least loaded eligible host when provisioning starts.</p> <p>Hosts are assigned when provisioning starts. You can queue launches while hosts are busy or preparing; each server waits for an eligible host and available capacity.</p>
{#if !loading && !hosts.length}<p><a href="/hosts" target="_blank" rel="noopener">Add a host</a>. Its runtime and latest T5 and T6 files will be prepared automatically.</p>{/if} {#if !loading && !hosts.length}<p><a href="/hosts" target="_blank" rel="noopener">Add a host</a>. Its runtime and latest T5 and T6 files will be prepared automatically.</p>{/if}
{#if !loading}{#each games as game} {#if !loading}{#each games as game}
<div class="game-files"> <div class="game-files">
<strong>{game === 't5' ? 'Black Ops I (T5)' : 'Black Ops II (T6)'} files</strong> <strong>{game === 't5' ? 'Black Ops I (T5)' : 'Black Ops II (T6)'} files</strong>
{#if latest[game]} {#if latest[game]}
<p>Latest upload · {$timeFormat.dateTime(latest[game]?.publishedAt ?? latest[game]?.createdAt)}</p> <p>Latest upload · {$timeFormat.dateTime(latest[game]?.publishedAt ?? latest[game]?.createdAt)}</p>
<p role="status">{cacheReady(game) ? 'Latest files ready on an eligible host.' : 'Waiting for a host to prepare the runtime and latest files automatically.'}</p> <p role="status">{cacheReady(game) ? 'Latest files ready on an eligible host.' : 'Launches can be queued now and will wait for a host to prepare the runtime and latest files.'}</p>
{:else} {:else}
<p>No files uploaded yet. <a href="/game-files" target="_blank" rel="noopener">Upload {game.toUpperCase()} files</a> to prepare your hosts.</p> <p>No files uploaded yet. <a href="/game-files" target="_blank" rel="noopener">Upload {game.toUpperCase()} files</a> to prepare your hosts.</p>
{/if} {/if}
@@ -24,17 +24,15 @@
const waitingForPort = (server: CollectionServer) => server.state === 'queued' && (server.stage.startsWith('Waiting for removal') || server.stage.startsWith('Waiting for port')); const waitingForPort = (server: CollectionServer) => server.state === 'queued' && (server.stage.startsWith('Waiting for removal') || server.stage.startsWith('Waiting for port'));
function stateLabel(server: CollectionServer) { function stateLabel(server: CollectionServer) {
if (waitingForPort(server)) return 'Waiting for removal'; if (waitingForPort(server)) return 'Waiting for removal';
if (server.state === 'queued' && server.queuePosition === 1) return 'Up next';
if (server.action === 'destroy' && server.state === 'done') return 'Removed'; if (server.action === 'destroy' && server.state === 'done') return 'Removed';
if (server.action === 'destroy' && server.state === 'running') return 'Removing'; if (server.action === 'destroy' && server.state === 'running') return 'Removing';
return labels[server.state] ?? server.state; return labels[server.state] ?? server.state;
} }
function queueHint(server: CollectionServer) { function queueHint(server: CollectionServer) {
if (waitingForPort(server)) return server.stage; const reason = server.stage || 'Waiting for a startup slot and available host capacity.';
if (server.queuePosition === 1) return 'Next in the launch queue. Starts when a place opens.'; return server.queuePosition && server.queuePosition > 1
if (server.queuePosition) return `${server.queuePosition - 1} queued server${server.queuePosition === 2 ? '' : 's'} ahead. Starts automatically when it is this lobby’s turn.`; ? `${reason} ${server.queuePosition - 1} queued server${server.queuePosition === 2 ? '' : 's'} ahead.` : reason;
return 'Waiting for its turn in the launch queue.';
} }
async function load() { async function load() {
@@ -90,11 +88,12 @@
{:else if firstQueued} {:else if firstQueued}
<aside class="waiting-note"> <aside class="waiting-note">
<span class="waiting-icon" aria-hidden="true">◷</span> <span class="waiting-icon" aria-hidden="true">◷</span>
<div><strong>{waitingForPort(firstQueued) ? 'Waiting for a removal to release capacity' : otherLaunches.length ? 'Waiting for an earlier launch' : firstQueued?.queuePosition && firstQueued.queuePosition > 1 ? 'Earlier servers are ahead in the queue' : 'Your launch is up next'}</strong> <div><strong>{waitingForPort(firstQueued) ? 'Waiting for a removal to release capacity' : 'Your collection is queued'}</strong>
<p>{queueHint(firstQueued)} Provisioning starts each server when its host and capacity are ready. Progress updates automatically.</p>
{#if otherLaunches.length} {#if otherLaunches.length}
<p>Your collection will start automatically as earlier launches finish. Currently launching:</p> <p>Other servers currently launching:</p>
<div class="other-launches">{#each otherLaunches as active (active.id)}<a href="/provisioning/{active.id}"><ServerIdentity name={active.server} /> <span aria-hidden="true">→</span></a>{/each}</div> <div class="other-launches">{#each otherLaunches as active (active.id)}<a href="/provisioning/{active.id}"><ServerIdentity name={active.server} /> <span aria-hidden="true">→</span></a>{/each}</div>
{:else}<p>{firstQueued ? queueHint(firstQueued) : 'Your servers will begin automatically.'} Progress updates automatically.</p>{/if} {/if}
</div> </div>
</aside> </aside>
{/if} {/if}
@@ -114,7 +113,7 @@
<div class="server-content"> <div class="server-content">
<div class="server-heading"><strong><HistoricalServerIdentity name={server.server} deletedAt={server.server_deleted_at} /></strong><Badge variant={needsAttention(server) ? 'danger' : server.state === 'done' ? 'online' : launching(server) ? 'info' : 'neutral'}>{stateLabel(server)}</Badge></div> <div class="server-heading"><strong><HistoricalServerIdentity name={server.server} deletedAt={server.server_deleted_at} /></strong><Badge variant={needsAttention(server) ? 'danger' : server.state === 'done' ? 'online' : launching(server) ? 'info' : 'neutral'}>{stateLabel(server)}</Badge></div>
<p class:error={!!server.error}>{server.error ?? (server.state === 'queued' ? queueHint(server) : server.state === 'done' ? server.action === 'destroy' ? server.stage : 'Startup checks passed.' : server.stage)}</p> <p class:error={!!server.error}>{server.error ?? (server.state === 'queued' ? queueHint(server) : server.state === 'done' ? server.action === 'destroy' ? server.stage : 'Startup checks passed.' : server.stage)}</p>
<div class="server-meta"><span>{server.action === 'destroy' ? 'Remove server' : `Create server · Port ${server.port}`}</span><span>{needsAttention(server) ? 'Review & recover' : 'View operation'} <span aria-hidden="true">→</span></span></div> <div class="server-meta"><span>{server.action === 'destroy' ? 'Remove server' : server.port === null ? 'Create server · Port not assigned yet' : `Create server · Port ${server.port}`}</span><span>{needsAttention(server) ? 'Review & recover' : 'View operation'} <span aria-hidden="true">→</span></span></div>
</div> </div>
</a> </a>
</li> </li>
@@ -79,23 +79,46 @@ describe('large native collection admission', () => {
expect(preview.ready).toBe(true); expect(preview.ready).toBe(true);
expect(preview.servers).toHaveLength(serverCount); expect(preview.servers).toHaveLength(serverCount);
expect(new Set(preview.servers.map(server => server.name)).size).toBe(serverCount); expect(new Set(preview.servers.map(server => server.name)).size).toBe(serverCount);
expect(new Set(preview.servers.map(server => server.port)).size).toBe(serverCount); expect(preview.servers.every(server => server.port === null)).toBe(true);
expect(db().prepare('SELECT * FROM ec_host_ports').all()).toHaveLength(0); expect(db().prepare('SELECT * FROM ec_host_ports').all()).toHaveLength(0);
const native = await import('../../src/lib/server/provisioning/native.js');
const validateSelection = vi.spyOn(native, 'nativeLaunchSelection');
manifestParses = 0; manifestParses = 0;
const result = queueCollection({ previewId: preview.id, const result = queueCollection({ previewId: preview.id,
keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name])) }); keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name])) });
const queueParses = manifestParses; const queueParses = manifestParses;
expect(result).toMatchObject({ ok: true, collectionId: expect.any(String) }); expect(result).toMatchObject({ ok: true, collectionId: expect.any(String) });
expect(queueCollection({ previewId: preview.id })).toEqual(result); expect(queueCollection({ previewId: preview.id })).toEqual(result);
expect(validateSelection).toHaveBeenCalledTimes(1);
expect(getCollection(result.collectionId!)?.collection).toMatchObject({ total: serverCount, queued: serverCount }); expect(getCollection(result.collectionId!)?.collection).toMatchObject({ total: serverCount, queued: serverCount });
const instances = db().prepare('SELECT host_id,library_id,port FROM native_instances ORDER BY port').all(); expect(db().prepare('SELECT host_id,library_id,port FROM native_instances').all()).toEqual([]);
expect(instances).toEqual(Array.from({ length: serverCount }, (_, index) => ({ expect(db().prepare('SELECT * FROM ec_host_ports').all()).toEqual([]);
host_id: hostId, library_id: libraryId, port: 28960 + index const intents = db().prepare('SELECT selection FROM native_creation_intents').all() as { selection: string }[];
}))); expect(intents).toHaveLength(serverCount);
expect(db().prepare('SELECT * FROM ec_host_ports').all()).toHaveLength(serverCount * 2); expect(intents.every(intent => JSON.parse(intent.selection).libraryId === libraryId)).toBe(true);
expect(db().prepare('SELECT * FROM server_collection_members').all()).toHaveLength(serverCount); expect(db().prepare('SELECT * FROM server_collection_members').all()).toHaveLength(serverCount);
expect(previewParses, 'Preview must use library metadata without reading the full manifest').toBe(0); expect(previewParses, 'Preview must use library metadata without reading the full manifest').toBe(0);
expect(queueParses, 'Queue admission must use library metadata without reading the full manifest').toBe(0); expect(queueParses, 'Queue admission must use library metadata without reading the full manifest').toBe(0);
}, 30_000); }, 30_000);
it.each(['newer game files', 'deleted pinned host'])('rejects %s after review before queueing any collection work', async change => {
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
const { db } = await import('../../src/lib/server/db/index.js');
const blueprint = saveBlueprint({ name: 'Reviewed native selection', entries: [{ game: 't5', mapSlug: 'theater', count: 2 }] });
const preview = previewCollection(blueprint.id, 'fill', { hostId, libraryId, image });
if (change === 'newer game files') {
const { createLibraryImport } = await import('../../src/lib/server/game-files/index.js');
const newer = createLibraryImport({ hostId, game: 't5' });
db().prepare("UPDATE game_file_versions SET state='ready',published_at=? WHERE id=?")
.run(new Date(Date.now() + 1000).toISOString(), newer.id);
} else db().prepare('UPDATE ec_hosts SET deleted_at=? WHERE id=?').run(new Date().toISOString(), hostId);
expect(() => queueCollection({ previewId: preview.id,
keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name])) }))
.toThrow(change === 'newer game files' ? /game files changed/ : /host was not found/);
expect(db().prepare('SELECT id FROM server_launch_collections').all()).toEqual([]);
expect(db().prepare('SELECT deployment_id FROM native_creation_intents').all()).toEqual([]);
expect(db().prepare('SELECT server_id FROM server_keys').all()).toEqual([]);
});
}); });
+105 -8
View File
@@ -12,10 +12,14 @@ vi.mock('../../src/lib/server/integrations/rcon.js', () => ({
let fixture: Fixture; let fixture: Fixture;
let hostId: string; let hostId: string;
let agentCredential: string;
let heartbeatInventory: Record<string, unknown>;
let releasePlacement: (() => void) | undefined;
const image = 'example.test/plutainer:test'; const image = 'example.test/plutainer:test';
beforeEach(async () => { beforeEach(async () => {
vi.resetModules(); vi.clearAllMocks(); vi.resetModules(); vi.clearAllMocks();
releasePlacement = undefined;
fixture = await newFixture([]); fixture = await newFixture([]);
vi.stubEnv('GAME_FILES_ROOT', join(fixture.root, 'libraries')); vi.stubEnv('GAME_FILES_ROOT', join(fixture.root, 'libraries'));
vi.stubEnv('PLUTAINER_IMAGE', image); vi.stubEnv('PLUTAINER_IMAGE', image);
@@ -31,7 +35,7 @@ beforeEach(async () => {
protocolVersion: 1, dockerRunning: true, directoriesWritable: true, protocolVersion: 1, dockerRunning: true, directoriesWritable: true,
storageAvailable: 1_000_000_000_000, runtimeImages: [image] storageAvailable: 1_000_000_000_000, runtimeImages: [image]
}; };
enrollHost(createEnrollment(hostId).token, inventory); agentCredential = enrollHost(createEnrollment(hostId).token, inventory).credential;
const { createLibraryImport, updateHostLibraryCaches } = await import('../../src/lib/server/game-files/index.js'); const { createLibraryImport, updateHostLibraryCaches } = await import('../../src/lib/server/game-files/index.js');
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const libraries = ['t5', 't6'].map(game => { const libraries = ['t5', 't6'].map(game => {
@@ -41,7 +45,8 @@ beforeEach(async () => {
return { id, state: 'ready' as const }; return { id, state: 'ready' as const };
}); });
updateHostLibraryCaches(hostId, libraries.map(library => ({ libraryVersionId: library.id, status: 'ready', manifestSha256: 'a'.repeat(64) }))); updateHostLibraryCaches(hostId, libraries.map(library => ({ libraryVersionId: library.id, status: 'ready', manifestSha256: 'a'.repeat(64) })));
heartbeat(hostId, { ...inventory, libraries }); heartbeatInventory = { ...inventory, libraries };
heartbeat(hostId, heartbeatInventory);
const { T6_ZOMBIE_MAPS } = await import('../../src/lib/map-aliases.js'); const { T6_ZOMBIE_MAPS } = await import('../../src/lib/map-aliases.js');
const now = new Date().toISOString(); const now = new Date().toISOString();
db().transaction(() => { db().transaction(() => {
@@ -68,6 +73,7 @@ beforeEach(async () => {
}); });
afterEach(async () => { afterEach(async () => {
releasePlacement?.();
await new Promise<void>(resolve => setImmediate(resolve)); await new Promise<void>(resolve => setImmediate(resolve));
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
await processProvisioning(); await processProvisioning();
@@ -78,7 +84,44 @@ afterEach(async () => {
}); });
describe('matching a native collection', () => { describe('matching a native collection', () => {
it('admits nine removals and ten creations without copying plugin bytes into every queued launch', async () => { it('places queued creations and completes all exact removals during later worker passes', async () => {
const { db } = await import('../../src/lib/server/db/index.js');
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
saveFleetSettings({ ...fleetSettings(), createConcurrency: 10, destroyConcurrency: 9 });
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
const { previewCollection, queueCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
const { T5_ZOMBIE_MAPS } = await import('../../src/lib/map-aliases.js');
const blueprint = saveBlueprint({ name: 'Replace nine with ten', entries: T5_ZOMBIE_MAPS.filter(map => map.slug !== 'zombietron')
.map(map => ({ game: 't5', mapSlug: map.slug, count: 1 })) });
const preview = previewCollection(blueprint.id, 'exact');
const queued = queueCollection({ previewId: preview.id,
keys: Object.fromEntries(preview.servers.map(server => [server.name!, `key-${server.name}`])) });
expect(queueCollection({ previewId: preview.id })).toEqual(queued);
expect(db().prepare('SELECT id FROM native_instances').all()).toHaveLength(9);
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toHaveLength(18);
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
await processProvisioning();
const operations = db().prepare("SELECT id,kind FROM ec_host_operations WHERE kind IN ('instance.create','instance.remove')")
.all() as { id: string; kind: string }[];
expect(operations.filter(operation => operation.kind === 'instance.create')).toHaveLength(10);
const removals = operations.filter(operation => operation.kind === 'instance.remove');
expect(removals).toHaveLength(9);
const { reportHostOperationStatus } = await import('../../src/lib/server/hosts/index.js');
for (const operation of removals) reportHostOperationStatus(hostId, operation.id, { state: 'completed', result: { exists: false } });
await processProvisioning();
const active = db().prepare('SELECT id,port FROM native_instances WHERE retired_at IS NULL').all() as { id: string; port: number }[];
expect(active).toHaveLength(10);
expect(new Set(active.map(instance => instance.port)).size).toBe(10);
for (const instance of active) expect(db().prepare('SELECT protocol,port FROM ec_host_ports WHERE instance_id=? ORDER BY protocol').all(instance.id))
.toEqual([{ protocol: 'tcp', port: instance.port }, { protocol: 'udp', port: instance.port }]);
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toHaveLength(20);
expect(getCollection(queued.collectionId!)?.servers.filter(server => server.action === 'destroy').map(server => server.state))
.toEqual(Array(9).fill('done'));
expect(db().prepare("SELECT id FROM ec_host_operations WHERE kind IN ('instance.create','instance.remove')").all()).toHaveLength(19);
});
it('accepts nine removals and ten unplaced creations before capacity checks or file loading', async () => {
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const { createScript, setScope, deleteScript } = await import('../../src/lib/server/fleet/scripts.js'); const { createScript, setScope, deleteScript } = await import('../../src/lib/server/fleet/scripts.js');
const content = Buffer.alloc(8 * 1024 ** 2, 42); const content = Buffer.alloc(8 * 1024 ** 2, 42);
@@ -91,6 +134,10 @@ describe('matching a native collection', () => {
const { nativeRequest } = await import('../../src/lib/server/backends/native.js'); const { nativeRequest } = await import('../../src/lib/server/backends/native.js');
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
const { POST } = await import('../../src/routes/api/deployment-collections/+server.js'); const { POST } = await import('../../src/routes/api/deployment-collections/+server.js');
const { load } = await import('../../src/routes/collections/launches/[id]/+page.server.js');
const { GET: collectionResponse } = await import('../../src/routes/api/deployment-collections/[id]/+server.js');
const { GET: fleetResponse } = await import('../../src/routes/api/servers/+server.js');
const { POST: agentHeartbeat } = await import('../../src/routes/api/agent/heartbeat/+server.js');
const { T5_ZOMBIE_MAPS } = await import('../../src/lib/map-aliases.js'); const { T5_ZOMBIE_MAPS } = await import('../../src/lib/map-aliases.js');
const blueprint = saveBlueprint({ name: 'Match ten T5 lobbies', entries: T5_ZOMBIE_MAPS.filter(map => map.slug !== 'zombietron') const blueprint = saveBlueprint({ name: 'Match ten T5 lobbies', entries: T5_ZOMBIE_MAPS.filter(map => map.slug !== 'zombietron')
.map(map => ({ game: 't5', mapSlug: map.slug, count: 1 })) }); .map(map => ({ game: 't5', mapSlug: map.slug, count: 1 })) });
@@ -102,6 +149,23 @@ describe('matching a native collection', () => {
.get(preview.id) as { bytes: number }; .get(preview.id) as { bytes: number };
expect(storedPreview.bytes).toBeLessThan(100_000); expect(storedPreview.bytes).toBeLessThan(100_000);
const capacity = await import('../../src/lib/server/hosts/capacity.js');
const placement = vi.spyOn(capacity, 'choosePlacement').mockImplementation(() => {
throw new Error('Placement is waiting for fresh host capacity');
});
const preparation = await import('../../src/lib/server/provisioning/launch-preparation.js');
const evaluate = preparation.evaluateNativePlacement;
let placementStarted!: () => void;
const started = new Promise<void>(resolve => { placementStarted = resolve; });
const gate = new Promise<void>(resolve => { releasePlacement = resolve; });
const evaluation = vi.spyOn(preparation, 'evaluateNativePlacement').mockImplementation(async options => {
placementStarted();
await gate;
return evaluate(options);
});
const scripts = await import('../../src/lib/server/fleet/scripts.js');
const fullVersion = vi.spyOn(scripts, 'getVersion');
const submit = (keys?: Record<string, string>) => POST({ request: new Request('http://localhost/api/deployment-collections', { const submit = (keys?: Record<string, string>) => POST({ request: new Request('http://localhost/api/deployment-collections', {
method: 'POST', body: JSON.stringify({ previewId: preview.id, keys }) }) }); method: 'POST', body: JSON.stringify({ previewId: preview.id, keys }) }) });
const response = await submit(Object.fromEntries(preview.servers.map(server => [server.name!, `key-${server.name}`]))); const response = await submit(Object.fromEntries(preview.servers.map(server => [server.name!, `key-${server.name}`])));
@@ -109,21 +173,54 @@ describe('matching a native collection', () => {
expect(response.status).toBe(202); expect(response.status).toBe(202);
expect(body).toMatchObject({ ok: true, collectionId: expect.any(String) }); expect(body).toMatchObject({ ok: true, collectionId: expect.any(String) });
expect(await (await submit()).json()).toEqual(body); expect(await (await submit()).json()).toEqual(body);
expect(load({ params: { id: body.collectionId } })).toEqual({});
// Exercise the real route and worker: admission must return before // Exercise the real route and worker: admission must return before
// loading launch payloads or inspecting the nine servers to remove. // loading launch payloads or inspecting the nine servers to remove.
expect(nativeRequest).not.toHaveBeenCalled(); expect(nativeRequest).not.toHaveBeenCalled();
expect(placement).not.toHaveBeenCalled();
expect(evaluation).not.toHaveBeenCalled();
expect(fullVersion).not.toHaveBeenCalled();
expect(db().prepare("SELECT id FROM ec_host_operations WHERE kind='instance.create'").all()).toEqual([]); expect(db().prepare("SELECT id FROM ec_host_operations WHERE kind='instance.create'").all()).toEqual([]);
for (const table of ['server_launch_previews', 'server_launch_recipes']) { expect(db().prepare('SELECT id FROM native_instances').all()).toHaveLength(9);
const rows = db().prepare(`SELECT length(recipe) AS bytes FROM ${table}`).all() as { bytes: number }[]; expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toHaveLength(18);
expect(rows).toHaveLength(10); expect(db().prepare('SELECT deployment_id FROM native_creation_intents').all()).toHaveLength(10);
expect(rows.every(row => row.bytes < 10_000)).toBe(true); expect(db().prepare('SELECT id FROM server_launch_previews').all()).toEqual([]);
} const rows = db().prepare('SELECT length(recipe) AS bytes FROM server_launch_recipes').all() as { bytes: number }[];
expect(rows).toHaveLength(10);
expect(rows.every(row => row.bytes < 10_000)).toBe(true);
const collection = getCollection(body.collectionId)!; const collection = getCollection(body.collectionId)!;
expect(collection.collection).toMatchObject({ mode: 'exact', total: 19 }); expect(collection.collection).toMatchObject({ mode: 'exact', total: 19 });
expect(collection.servers.filter(server => server.action === 'destroy')).toHaveLength(9); expect(collection.servers.filter(server => server.action === 'destroy')).toHaveLength(9);
expect(collection.servers.filter(server => server.action === 'create')).toHaveLength(10); expect(collection.servers.filter(server => server.action === 'create')).toHaveLength(10);
const { nativeDeployment } = await import('../../src/lib/server/backends/native.js');
for (const server of collection.servers.filter(server => server.action === 'create')) {
expect(nativeDeployment(server.id)).toBe(true);
expect(getDeployment(server.id)?.backend).toBe('native');
}
await new Promise<void>(resolve => setImmediate(resolve)); await new Promise<void>(resolve => setImmediate(resolve));
await processProvisioning({ wait: false });
await started;
expect(evaluation).toHaveBeenCalled();
expect(placement).not.toHaveBeenCalled();
expect(db().prepare("SELECT id FROM ec_host_operations WHERE kind='instance.create'").all()).toEqual([]);
expect(getCollection(body.collectionId)?.collection.total).toBe(19);
const collectionPage = collectionResponse({ params: { id: body.collectionId } });
expect(collectionPage.status).toBe(200);
expect((await collectionPage.json()).servers).toHaveLength(19);
const fleet = fleetResponse();
expect(fleet.status).toBe(200);
expect((await fleet.json()).servers).toHaveLength(9);
const heartbeatResponse = await agentHeartbeat({ request: new Request('https://controller.test/api/agent/heartbeat', {
method: 'POST', headers: { authorization: `Bearer ${agentCredential}` }, body: JSON.stringify({ inventory: heartbeatInventory })
}) });
expect(heartbeatResponse.status).toBe(200);
expect(await heartbeatResponse.json()).toMatchObject({ host: { id: hostId, connected: true }, heartbeatSeconds: 15 });
// All three endpoints complete while the background placement is still paused.
expect(placement).not.toHaveBeenCalled();
expect(db().prepare("SELECT id FROM ec_host_operations WHERE kind='instance.create'").all()).toEqual([]);
placement.mockRestore(); fullVersion.mockRestore();
releasePlacement!();
await processProvisioning(); await processProvisioning();
const dispatched = db().prepare(`SELECT n.deployment_id,o.id FROM native_provisioning n const dispatched = db().prepare(`SELECT n.deployment_id,o.id FROM native_provisioning n
JOIN ec_host_operations o ON o.id=n.operation_id WHERE o.kind='instance.create'`).all() as { deployment_id: string; id: string }[]; JOIN ec_host_operations o ON o.id=n.operation_id WHERE o.kind='instance.create'`).all() as { deployment_id: string; id: string }[];
+1
View File
@@ -481,6 +481,7 @@ describe('server creation map selection', () => {
expect(getDeployment(id)?.deployment).toMatchObject({ state: 'awaiting-health', stage: 'Waiting for RCON or fresh telemetry health: timeout' }); expect(getDeployment(id)?.deployment).toMatchObject({ state: 'awaiting-health', stage: 'Waiting for RCON or fresh telemetry health: timeout' });
for (const map of [null, 'zombie_moon']) { for (const map of [null, 'zombie_moon']) {
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: true, state: 'running' }); vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: true, state: 'running' });
if (created.port === null) throw new Error('A provisioned legacy server must have a port.');
vi.mocked(rconStatusResult).mockResolvedValueOnce({ ok: true, endpoint: { host: 'localhost', port: created.port }, rawReply: '', status: { players: 0, map, gametype: null, roster: [] } }); vi.mocked(rconStatusResult).mockResolvedValueOnce({ ok: true, endpoint: { host: 'localhost', port: created.port }, rawReply: '', status: { players: 0, map, gametype: null, roster: [] } });
await processCreations(); await processCreations();
expect(getDeployment(id)?.deployment).toMatchObject({ state: 'awaiting-health', finished_at: null, stage: expect.stringContaining('expected zombie_cosmodrome') }); expect(getDeployment(id)?.deployment).toMatchObject({ state: 'awaiting-health', finished_at: null, stage: expect.stringContaining('expected zombie_cosmodrome') });
+104
View File
@@ -0,0 +1,104 @@
import { setImmediate } from 'node:timers/promises';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { newFixture, type Fixture } from '../helpers/fixture.js';
let fixture: Fixture;
let hostId: string;
const GiB = 1024 ** 3;
const epoch = Date.parse('2026-10-05T12:00:00Z');
const inventory = { protocolVersion: 1, dockerRunning: true, directoriesWritable: true, storageAvailable: 100 * GiB,
cpuCount: 16, cpuUsagePercent: 0, memoryTotal: 64 * GiB, memoryAvailable: 60 * GiB,
runtimeImages: ['image'], libraries: [{ id: 't5-library', state: 'ready' }, { id: 't6-library', state: 'ready' }] };
const t5 = { libraryId: 't5-library', image: 'image', automatic: true };
const t6 = { libraryId: 't6-library', image: 'image', automatic: true };
async function settle() { await setImmediate(); await setImmediate(); }
beforeEach(async () => {
vi.resetModules();
fixture = await newFixture([]);
vi.useFakeTimers({ toFake: ['Date'] }); vi.setSystemTime(epoch);
const hosts = await import('../../src/lib/server/hosts/index.js');
hostId = hosts.createHost({ name: 'Availability host', portMin: 28960, portMax: 28962 }).id;
hosts.enrollHost(hosts.createEnrollment(hostId).token, inventory);
hosts.heartbeat(hostId, inventory);
await settle();
});
afterEach(async () => { vi.restoreAllMocks(); vi.useRealTimers(); await fixture.cleanup(); });
describe('cached launch availability', () => {
it('reads cached estimates without placement, historical scans, or repeated database reads', async () => {
const { db } = await import('../../src/lib/server/db/index.js');
const capacity = await import('../../src/lib/server/hosts/capacity.js');
const { launchAvailability } = await import('../../src/lib/server/hosts/availability.js');
const placement = vi.spyOn(capacity, 'choosePlacement');
const prepare = vi.spyOn(db(), 'prepare');
for (let count = 0; count < 100; count++) {
expect(launchAvailability([t5, t6]).availability).toMatchObject({
updatedAt: new Date(epoch).toISOString(), requested: 2, estimatedAvailable: 2, portsAvailable: 3
});
}
expect(prepare).not.toHaveBeenCalled();
expect(placement).not.toHaveBeenCalled();
});
it('shares CPU, memory and ports across games and refreshes measurements on heartbeat', async () => {
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
const { launchAvailability } = await import('../../src/lib/server/hosts/availability.js');
heartbeat(hostId, { ...inventory, memoryAvailable: GiB + 500_000_000 });
const limited = launchAvailability([t5, t6]);
expect(limited.availability).toMatchObject({ requested: 2, estimatedAvailable: 1, portsAvailable: 3,
cpuCoresAvailable: 15.2, memoryBytesAvailable: GiB });
expect(limited.warnings.join(' ')).toContain('1 of 2');
vi.setSystemTime(epoch + 15_000);
heartbeat(hostId, { ...inventory, memoryAvailable: 2 * GiB + 500_000_000 });
expect(launchAvailability([t5, t6]).availability).toMatchObject({ estimatedAvailable: 2,
memoryBytesAvailable: 2 * GiB, updatedAt: new Date(epoch + 15_000).toISOString() });
heartbeat(hostId, { ...inventory, cpuUsagePercent: 100 });
expect(launchAvailability([t5]).availability.estimatedAvailable).toBe(0);
});
it('refreshes port reservations and startup queue snapshots after lifecycle events', async () => {
const { reserveHostPort, releaseHostPorts } = await import('../../src/lib/server/hosts/index.js');
const { launchAvailability } = await import('../../src/lib/server/hosts/availability.js');
const { beginDeployment, deploymentStage } = await import('../../src/lib/server/provisioning/history.js');
const { db } = await import('../../src/lib/server/db/index.js');
const limit = launchAvailability([t5]).availability.startupLimit!;
reserveHostPort(hostId, 'reserved', { port: 28960, protocol: 'udp' });
reserveHostPort(hostId, 'reserved', { port: 28960, protocol: 'tcp' });
await settle();
expect(launchAvailability([t5, t6, t5]).availability).toMatchObject({ portsAvailable: 2, estimatedAvailable: 2 });
const active = beginDeployment('t5_zombie_theater_1', 'create');
const queued = beginDeployment('t6_zombie_tranzit_1', 'create');
db().prepare("UPDATE server_deployments SET state='queued' WHERE id=?").run(queued);
deploymentStage(queued, 'Awaiting placement', { checkpoint: { key: 'queued', label: 'Queued for placement' } });
await settle();
expect(launchAvailability([t5]).availability).toMatchObject({ queued: 1, startupSlots: Math.max(0, limit - 1) });
db().prepare("UPDATE server_deployments SET state='done' WHERE id=?").run(active);
releaseHostPorts('reserved');
await settle();
expect(launchAvailability([t5]).availability).toMatchObject({ portsAvailable: 3, startupSlots: limit });
});
it('stops counting a stale host without presenting old measurements as current capacity', async () => {
const { launchAvailability } = await import('../../src/lib/server/hosts/availability.js');
vi.setSystemTime(epoch + 91_000);
const result = launchAvailability([t5]);
expect(result.availability).toMatchObject({ updatedAt: new Date(epoch).toISOString(), estimatedAvailable: 0, portsAvailable: 0 });
expect(result.warnings.join(' ')).toContain('Waiting for an eligible host');
});
it('keeps heartbeats working while provisioning schema is unavailable and refreshes after initialization', async () => {
const { db } = await import('../../src/lib/server/db/index.js');
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
const { launchAvailability } = await import('../../src/lib/server/hosts/availability.js');
const errors = vi.spyOn(console, 'error');
db().exec('ALTER TABLE native_provisioning RENAME TO unavailable_native_provisioning');
expect(() => heartbeat(hostId, inventory)).not.toThrow();
expect(launchAvailability([t5]).availability.updatedAt).toBeNull();
await settle();
expect(errors).not.toHaveBeenCalled();
db().exec('ALTER TABLE unavailable_native_provisioning RENAME TO native_provisioning');
heartbeat(hostId, inventory);
expect(launchAvailability([t5]).availability).toMatchObject({ estimatedAvailable: 1, portsAvailable: 3 });
});
});
+41
View File
@@ -77,6 +77,9 @@ describe('host deletion', () => {
const queued = hosts.deleteHost(hosts.host.id); const queued = hosts.deleteHost(hosts.host.id);
expect(queued.deleted).toBe(false); expect(queued.deleted).toBe(false);
expect(hosts.getHost(hosts.host.id)?.decommission?.stage).toBe('queued'); expect(hosts.getHost(hosts.host.id)?.decommission?.stage).toBe('queued');
const { nativeLaunchSelection } = await import('../../src/lib/server/provisioning/native.js');
expect(() => nativeLaunchSelection({ hostId: hosts.host.id, libraryId: 'version', image: 'plutainer:test' }, 't5'))
.toThrow('The selected host is being removed');
expect(hosts.authenticateCredential(hosts.credentials.credential)?.id).toBe(hosts.host.id); expect(hosts.authenticateCredential(hosts.credentials.credential)?.id).toBe(hosts.host.id);
await finishCleanup(hosts); await finishCleanup(hosts);
@@ -120,6 +123,44 @@ describe('host deletion', () => {
expect(hosts.authenticateCredential(hosts.credentials.credential)?.id).toBe(hosts.host.id); expect(hosts.authenticateCredential(hosts.credentials.credential)?.id).toBe(hosts.host.id);
}); });
it('blocks an accepted unplaced pinned launch until it is cancelled', async () => {
const hosts = await enrolled();
const { queueNativeCreationIntent } = await import('../../src/lib/server/provisioning/native.js');
const { cancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
const { db } = await import('../../src/lib/server/db/index.js');
const queued = queueNativeCreationIntent({ server: 't5_zombie_theater_1', game: 't5', mapSlug: 'theater',
plutoKey: 'saved-key', selection: { hostId: hosts.host.id, libraryId: 'library', image: 'plutainer:test' } });
expect(queued.ok).toBe(true);
if (!queued.ok) throw new Error(queued.message);
expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]);
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
expect(hosts.hostDeletionStatus(hosts.host.id)).toEqual({ allowed: false,
blockers: [expect.stringContaining('Cancel the queued server launches pinned to this host')] });
expect(() => hosts.deleteHost(hosts.host.id)).toThrow(expect.objectContaining({ status: 409 }));
expect(hosts.getHost(hosts.host.id)?.decommission).toBeNull();
expect(cancelCreation(queued.deploymentId!)).toBeTruthy();
expect(hosts.hostDeletionStatus(hosts.host.id)).toEqual({ allowed: true, blockers: [] });
await finishCleanup(hosts);
expect(hosts.getHost(hosts.host.id)).toBeNull();
});
it('does not pin a host for automatic launches or launches assigned elsewhere', async () => {
const hosts = await enrolled();
const other = hosts.createHost({ name: 'Other host' });
const { queueNativeCreationIntent } = await import('../../src/lib/server/provisioning/native.js');
for (const [index, selection] of [
{ hostId: hosts.host.id, automatic: true },
{ hostId: other.id, automatic: false }
].entries()) {
expect(queueNativeCreationIntent({ server: `t5_zombie_theater_${index + 1}`, game: 't5', mapSlug: 'theater',
plutoKey: 'saved-key', selection: { ...selection, libraryId: 'library', image: 'plutainer:test' } }).ok).toBe(true);
}
expect(hosts.hostDeletionStatus(hosts.host.id)).toEqual({ allowed: true, blockers: [] });
expect(hosts.hostDeletionStatus(other.id).allowed).toBe(false);
await finishCleanup(hosts);
expect(hosts.getHost(hosts.host.id)).toBeNull();
});
it('blocks residual port reservations independently of live instances', async () => { it('blocks residual port reservations independently of live instances', async () => {
const hosts = await enrolled(); const hosts = await enrolled();
hosts.reserveHostPort(hosts.host.id, 'unresolved-instance'); hosts.reserveHostPort(hosts.host.id, 'unresolved-instance');
+62
View File
@@ -243,6 +243,68 @@ describe('automatic host preparation', () => {
expect(libraries.requireReadyLibrary(id, old.id).id).toBe(old.id); expect(libraries.requireReadyLibrary(id, old.id).id).toBe(old.id);
}); });
it.each([false, true])('prepares queued runtime and library versions after newer publication (automatic placement: %s)', async automatic => {
const old = await publish();
const id = enroll({ runtimeImages: [runtime] });
const other = enroll({ runtimeImages: [runtime] });
const image = 'ghcr.io/example/plutainer:queued-version';
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
for (let index = 0; index < 2; index++) {
expect(queueNativeCreation({ game: 't5', gameMode: 'zombie', mapSlug: 'theater', libraryId: old.id,
hostId: automatic ? undefined : id, image, plutoKey: 'saved-key' }).ok).toBe(true);
}
const latest = await publish();
const status = preparation.hostPreparationStatus(hosts.getHost(id)!);
expect(status.items.filter(item => item.label === image)).toHaveLength(1);
expect(status.items.filter(item => item.label.includes('Queued version'))).toHaveLength(1);
const otherStatus = preparation.hostPreparationStatus(hosts.getHost(other)!);
expect(otherStatus.items.some(item => item.label === image)).toBe(automatic);
expect(otherStatus.items.some(item => item.label.includes('Queued version'))).toBe(automatic);
expect(operations(id)).toEqual([]);
preparation.reconcileHostPreparation(id);
const pull = operations(id)[0];
expect(pull).toMatchObject({ kind: 'runtime.pull', payload: { image } });
vi.setSystemTime(Date.now() + 1);
hosts.reportHostOperation(id, pull.id, { state: 'completed' });
heartbeat(id, { runtimeImages: [runtime, image] });
preparation.reconcileHostPreparation(id);
const download = operations(id)[0];
expect(download).toMatchObject({ kind: 'library.download', payload: { libraryVersionId: old.id, automatic: true } });
preparation.reconcileHostPreparation(id);
expect(hosts.getHostOperation(download.id)?.state).toBe('pending');
expect(operations(id)).toHaveLength(2);
verified(id, old);
vi.setSystemTime(Date.now() + 1);
hosts.reportHostOperation(id, download.id, { state: 'completed' });
heartbeat(id, { runtimeImages: [runtime, image], libraries: [{ id: old.id, state: 'ready' }] });
preparation.reconcileHostPreparation(id);
expect(operations(id)[0]).toMatchObject({ kind: 'library.download', payload: { libraryVersionId: latest.id } });
});
it('retains an undelivered library download for an accepted intent until that intent is cancelled', async () => {
const old = await publish();
const id = enroll({ runtimeImages: [runtime] });
preparation.reconcileHostPreparation(id);
const download = operations(id)[0];
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
const launch = queueNativeCreation({ game: 't5', gameMode: 'zombie', mapSlug: 'theater', libraryId: old.id,
hostId: id, image: runtime, plutoKey: 'saved-key' });
expect(launch.ok).toBe(true);
if (!launch.ok) throw new Error(launch.message);
const latest = await publish();
preparation.reconcileHostPreparation(id);
expect(hosts.getHostOperation(download.id)?.state).toBe('pending');
expect(operations(id)).toHaveLength(1);
const { cancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
expect(cancelCreation(launch.deploymentId)).toEqual(expect.any(String));
preparation.reconcileHostPreparation(id);
expect(hosts.getHostOperation(download.id)).toMatchObject({ state: 'failed', error: expect.stringContaining('Superseded') });
expect(operations(id)[0]).toMatchObject({ kind: 'library.download', payload: { libraryVersionId: latest.id } });
});
it('finishes a delivered download before following a newer upload under a new operation ID', async () => { it('finishes a delivered download before following a newer upload under a new operation ID', async () => {
const old = await publish(); const old = await publish();
const id = enroll({ runtimeImages: [runtime] }); const id = enroll({ runtimeImages: [runtime] });
+197
View File
@@ -0,0 +1,197 @@
import Database from 'better-sqlite3';
import { createHash } from 'node:crypto';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import schema from '../../src/lib/server/db/schema.sql?raw';
import { newFixture, type Fixture } from '../helpers/fixture.js';
let fixture: Fixture;
const server = 't5_zombie_theater_1';
beforeEach(async () => {
vi.resetModules();
fixture = await newFixture([]);
vi.stubEnv('KOMODO_ADDRESS', ''); vi.stubEnv('KOMODO_API_KEY', ''); vi.stubEnv('KOMODO_API_SECRET', '');
vi.stubEnv('RCON_PASSWORD', '');
});
afterEach(async () => { vi.restoreAllMocks(); await fixture.cleanup(); vi.unstubAllEnvs(); });
async function intent() {
const { db } = await import('../../src/lib/server/db/index.js');
const { ensureHostSchema } = await import('../../src/lib/server/hosts/schema.js');
ensureHostSchema(db());
const { beginDeployment } = await import('../../src/lib/server/provisioning/history.js');
const id = beginDeployment(server, 'create');
db().prepare("UPDATE server_deployments SET state='queued' WHERE id=?").run(id);
db().prepare('INSERT INTO native_creation_intents(deployment_id,selection) VALUES(?,?)')
.run(id, JSON.stringify({ automatic: true, libraryId: 'library', image: 'plutainer:test' }));
db().prepare(`INSERT INTO server_creation_jobs(deployment_id,server,port,game_mode,map_slug,expected_map)
VALUES(?,?,NULL,'zombie','theater','zombie_theater')`).run(id, server);
return id;
}
describe('native creation intents before placement', () => {
it('retains its identity and native history across restart without falling into legacy provisioning', async () => {
const id = await intent();
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
const { getDeployment, interruptDeployments } = await import('../../src/lib/server/provisioning/history.js');
const { nativeDeployment, unplacedNativeCreation } = await import('../../src/lib/server/backends/native.js');
const { nextLobbyNumber, processCreations } = await import('../../src/lib/server/provisioning/instances.js');
const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js');
expect(nativeDeployment(id)).toBe(true);
expect(unplacedNativeCreation(server)).toBe(id);
expect(nextLobbyNumber('t5_zombie_theater')).toBe(2);
expect(requestLifecycle(server, 'stop')).toMatchObject({ ok: false, status: 409 });
closeDb();
interruptDeployments();
await processCreations();
expect(getDeployment(id)).toMatchObject({ backend: 'native', canCancelCreation: true, agentOperation: null,
deployment: { state: 'queued', server_exists: 0 } });
expect(db().prepare('SELECT port FROM server_creation_jobs WHERE deployment_id=?').get(id)).toEqual({ port: null });
expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]);
expect(db().prepare('SELECT deployment_id FROM native_provisioning').all()).toEqual([]);
});
it('cancels unplaced work locally once, retains keys, and frees its logical server name', async () => {
const id = await intent();
const { db } = await import('../../src/lib/server/db/index.js');
const { saveServerKey, getServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
const { cancelCreation, canCancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
const { getDeployment } = await import('../../src/lib/server/provisioning/history.js');
const { nativeDeployment, unplacedNativeCreation } = await import('../../src/lib/server/backends/native.js');
const { nextLobbyNumber } = await import('../../src/lib/server/provisioning/instances.js');
saveServerKey(server, 'saved-key');
expect(canCancelCreation(id)).toBe(true);
const cleanup = cancelCreation(id)!;
expect(cleanup).toEqual(expect.any(String));
expect(cancelCreation(id)).toBe(cleanup);
expect(canCancelCreation(id)).toBe(false);
expect(nativeDeployment(cleanup)).toBe(true);
expect(getDeployment(id)).toMatchObject({ backend: 'native', cancellation: { state: 'done' }, deployment: { state: 'cancelled' } });
expect(getDeployment(cleanup)).toMatchObject({ backend: 'native', deployment: { state: 'done', server_deleted_at: expect.any(String) } });
expect(unplacedNativeCreation(server)).toBeNull();
expect(nextLobbyNumber('t5_zombie_theater')).toBe(1);
expect(getServerKey(server)).toBe('saved-key');
expect(db().prepare('SELECT id FROM ec_host_operations').all()).toEqual([]);
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
});
it('will not cancel an unplaced historical intent against a replacement server', async () => {
const id = await intent();
const { db } = await import('../../src/lib/server/db/index.js');
db().prepare("INSERT INTO servers(name,komodo_name) VALUES(?,'replacement')").run(server);
const { cancelCreation, canCancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
expect(canCancelCreation(id)).toBe(false);
expect(cancelCreation(id)).toBeNull();
expect(db().prepare('SELECT name FROM servers WHERE name=?').get(server)).toEqual({ name: server });
expect(db().prepare('SELECT cleanup_id FROM native_creation_cancellations').all()).toEqual([]);
});
it.each([false, true])('does not adopt a replacement instance during placement (previously bound: %s)', async previouslyBound => {
const id = await intent();
const { db } = await import('../../src/lib/server/db/index.js');
const { createHost } = await import('../../src/lib/server/hosts/index.js');
const { captureRecipePlan, recipeStorage } = await import('../../src/lib/server/fleet/launch-recipes.js');
const hostId = createHost({ name: 'Replacement host' }).id;
const insert = db().prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at,retired_at)
VALUES(?,?,?,'library','plutainer:test',28960,?,?)`);
if (previouslyBound) {
insert.run('original-instance', server, hostId, '2025-01-01', '2025-01-02');
db().prepare('INSERT INTO native_provisioning(deployment_id,instance_id,generation) VALUES(?,?,1)').run(id, 'original-instance');
}
insert.run('replacement-instance', server, hostId, '2025-01-02', null);
const before = db().prepare('SELECT * FROM native_instances ORDER BY id').all();
const stored = recipeStorage(captureRecipePlan('t5', 'theater'));
db().prepare('INSERT INTO server_launch_recipes(deployment_id,recipe,recipe_summary) VALUES(?,?,?)')
.run(id, stored.recipe, stored.recipeSummary);
const preparation = await import('../../src/lib/server/provisioning/launch-preparation.js');
const placement = vi.spyOn(preparation, 'evaluateNativePlacement').mockRejectedValue(new Error('Unexpected placement'));
const { processNativeProvisioning } = await import('../../src/lib/server/provisioning/native.js');
await processNativeProvisioning(() => null);
expect(placement).not.toHaveBeenCalled();
expect(db().prepare('SELECT state,error FROM server_deployments WHERE id=?').get(id))
.toEqual({ state: 'failed', error: 'Original instance was retired or replaced; no command was sent.' });
expect(db().prepare('SELECT * FROM native_instances ORDER BY id').all()).toEqual(before);
expect(db().prepare('SELECT id FROM ec_host_operations').all()).toEqual([]);
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
expect(db().prepare('SELECT port FROM server_creation_jobs WHERE deployment_id=?').get(id)).toEqual({ port: null });
});
it('starts work on a ready host when an older pinned intent is waiting for an offline host', async () => {
vi.stubEnv('GAME_FILES_ROOT', join(fixture.root, 'libraries'));
const image = 'plutainer:test';
const hosts = await import('../../src/lib/server/hosts/index.js');
const offline = hosts.createHost({ name: 'Offline host' });
const ready = hosts.createHost({ name: 'Ready host' });
const inventory = { protocolVersion: 1, dockerRunning: true, directoriesWritable: true,
cpuCount: 16, cpuUsagePercent: 5, memoryTotal: 32 * 1024 ** 3, memoryAvailable: 30 * 1024 ** 3,
storageAvailable: 100_000_000_000, runtimeImages: [image] };
hosts.enrollHost(hosts.createEnrollment(ready.id).token, inventory);
const libraries = await import('../../src/lib/server/game-files/index.js');
const imported = libraries.createLibraryImport({ hostId: ready.id, game: 't5' });
const bytes = Buffer.from('verified base game');
const file = { path: 'zone/base.ff', size: bytes.length, sha256: createHash('sha256').update(bytes).digest('hex') };
libraries.setImportManifest(ready.id, imported.id, [file]);
await libraries.uploadImportFile(ready.id, imported.id, file.path, new Response(bytes).body);
const library = await libraries.publishImport(ready.id, imported.id);
libraries.updateHostLibraryCaches(ready.id, [{ libraryVersionId: library.id, status: 'ready', manifestSha256: library.manifestSha256 }]);
hosts.heartbeat(ready.id, { ...inventory, libraries: [{ id: library.id, state: 'ready' }] });
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
saveFleetSettings({ ...fleetSettings(), createConcurrency: 1 });
const { queueNativeCreation, processNativeProvisioning } = await import('../../src/lib/server/provisioning/native.js');
const input = { game: 't5' as const, gameMode: 'zombie' as const, mapSlug: 'theater', libraryId: library.id, image, plutoKey: 'saved-key' };
const blocked = queueNativeCreation({ ...input, hostId: offline.id });
const available = queueNativeCreation({ ...input, hostId: ready.id });
expect(blocked.ok && available.ok).toBe(true);
if (!blocked.ok || !available.ok) throw new Error('Could not queue launch fixtures.');
await processNativeProvisioning(() => null);
const { db } = await import('../../src/lib/server/db/index.js');
expect(db().prepare('SELECT state FROM server_deployments WHERE id=?').get(blocked.deploymentId)).toEqual({ state: 'queued' });
expect(db().prepare('SELECT state FROM server_deployments WHERE id=?').get(available.deploymentId)).toEqual({ state: 'running' });
expect(db().prepare("SELECT host_id FROM ec_host_operations WHERE kind='instance.create'").all()).toEqual([{ host_id: ready.id }]);
expect(db().prepare('SELECT server,host_id FROM native_instances WHERE retired_at IS NULL').all())
.toEqual([{ server: available.name, host_id: ready.id }]);
expect(db().prepare('SELECT port FROM server_creation_jobs WHERE deployment_id=?').get(blocked.deploymentId)).toEqual({ port: null });
});
it('handles exact-match removal of an unplaced launch as local cancellation', async () => {
const id = await intent();
const { destroyServer } = await import('../../src/lib/server/provisioning/instances.js');
const { getDeployment } = await import('../../src/lib/server/provisioning/history.js');
const { db } = await import('../../src/lib/server/db/index.js');
const removal = destroyServer(server, 'blueprint-exact');
expect(removal).toMatchObject({ ok: true, deploymentId: expect.any(String) });
expect(getDeployment(id)?.deployment.state).toBe('cancelled');
expect(getDeployment(removal.deploymentId!)?.deployment.state).toBe('done');
expect(db().prepare('SELECT id FROM ec_host_operations').all()).toEqual([]);
});
it('migrates existing port reservations without losing rows, references, indexes, or triggers', async () => {
const legacy = new Database(fixture.dbPath);
try {
legacy.exec(schema.replace(/CREATE TABLE IF NOT EXISTS server_creation_jobs \([\s\S]*?\n\);/,
table => table.replace(' port INTEGER,', ' port INTEGER NOT NULL,')));
legacy.exec(`INSERT INTO server_deployments(id,server,action,state,stage,started_at,triggered_by)
VALUES('existing','old-server','create','queued','Queued','2025-01-01','fixture');
INSERT INTO server_creation_jobs(rowid,deployment_id,server,port,game_mode,map_slug,expected_map)
VALUES(8,'existing','old-server',28960,'zombie','theater','zombie_theater');
CREATE INDEX fixture_creation_port ON server_creation_jobs(port);
CREATE TABLE fixture_creation_reference(id TEXT REFERENCES server_creation_jobs(deployment_id));
INSERT INTO fixture_creation_reference(id) VALUES('existing');
CREATE TABLE fixture_creation_audit(port INTEGER);
CREATE TRIGGER fixture_creation_update AFTER UPDATE OF port ON server_creation_jobs
BEGIN INSERT INTO fixture_creation_audit(port) VALUES(NEW.port); END;`);
} finally { legacy.close(); }
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
for (let open = 0; open < 2; open++) {
const handle = db();
expect(handle.prepare('SELECT rowid,deployment_id,port FROM server_creation_jobs').get())
.toEqual({ rowid: 8, deployment_id: 'existing', port: 28960 });
expect(handle.prepare('SELECT id FROM fixture_creation_reference').get()).toEqual({ id: 'existing' });
expect(handle.prepare("SELECT name FROM sqlite_master WHERE name IN ('fixture_creation_port','fixture_creation_update')").all()).toHaveLength(2);
expect(handle.pragma('foreign_key_check')).toEqual([]);
closeDb();
}
db().prepare('UPDATE server_creation_jobs SET port=NULL WHERE deployment_id=?').run('existing');
expect(db().prepare('SELECT port FROM fixture_creation_audit').all()).toEqual([{ port: null }]);
});
});
@@ -83,6 +83,33 @@ async function pausePreparation() {
} }
describe('asynchronous native launch preparation', () => { describe('asynchronous native launch preparation', () => {
it('does not place an unsubmitted intent cancelled while host ranking is pending', async () => {
const job = await queue();
const preparation = await import('../../src/lib/server/provisioning/launch-preparation.js');
const evaluate = preparation.evaluateNativePlacement;
let started!: () => void;
let release!: () => void;
const ready = new Promise<void>(resolve => { started = resolve; });
const gate = new Promise<void>(resolve => { release = resolve; });
releases.push(release);
vi.spyOn(preparation, 'evaluateNativePlacement').mockImplementationOnce(async input => {
const result = await evaluate(input);
started(); await gate;
return result;
});
const pass = runPass();
await ready;
const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
expect(nativeInstance(job.name)).toBeNull();
const { cancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
expect(cancelCreation(job.deploymentId)).toEqual(expect.any(String));
release(); await pass;
expect(nativeInstance(job.name)).toBeNull();
expect(await operationFor(job.deploymentId)).toBeUndefined();
const { db } = await import('../../src/lib/server/db/index.js');
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
});
it('keeps controller timers and launch details responsive while preparing multiple large plugins', async () => { it('keeps controller timers and launch details responsive while preparing multiple large plugins', async () => {
const { createScript } = await import('../../src/lib/server/fleet/scripts.js'); const { createScript } = await import('../../src/lib/server/fleet/scripts.js');
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
+141 -34
View File
@@ -43,10 +43,11 @@ async function prepareLibrary(game: 't5' | 't6' = 't5') {
const { heartbeat } = await import('../../src/lib/server/hosts/index.js'); heartbeat(hostId, inventory()); const { heartbeat } = await import('../../src/lib/server/hosts/index.js'); heartbeat(hostId, inventory());
return ready; return ready;
} }
async function queue(game: 't5' | 't6' = 't5') { async function queue(game: 't5' | 't6' = 't5', placement: 'pinned' | 'automatic' = 'pinned') {
await prepareLibrary(game); remote.map = game === 't5' ? 'zombie_theater' : 'zm_transit'; await prepareLibrary(game); remote.map = game === 't5' ? 'zombie_theater' : 'zm_transit';
const { queueCreateServer } = await import('../../src/lib/server/provisioning/instances.js'); const { queueCreateServer } = await import('../../src/lib/server/provisioning/instances.js');
const result = queueCreateServer({ game, gameMode: 'zombie', mapSlug: game === 't5' ? 'theater' : 'town', hostId, libraryId, image, plutoKey: 'saved-key' }); const result = queueCreateServer({ game, gameMode: 'zombie', mapSlug: game === 't5' ? 'theater' : 'town',
hostId: placement === 'pinned' ? hostId : undefined, libraryId, image, plutoKey: 'saved-key' });
if (!result.ok) throw new Error(result.message); if (!result.ok) throw new Error(result.message);
return result; return result;
} }
@@ -73,6 +74,44 @@ async function detail(id: string) {
} }
describe('native instance provisioning', () => { describe('native instance provisioning', () => {
it('recognizes an unplaced native intent and resumes it once after a controller restart', async () => {
const job = await queue();
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
const { nativeDeployment, nativeInstance } = await import('../../src/lib/server/backends/native.js');
const { getDeployment } = await import('../../src/lib/server/provisioning/history.js');
expect(job.port).toBeNull();
expect(nativeDeployment(job.deploymentId)).toBe(true);
expect(nativeInstance(job.name)).toBeNull();
expect(getDeployment(job.deploymentId)).toMatchObject({ backend: 'native', canCancelCreation: true });
expect(db().prepare('SELECT deployment_id FROM native_creation_intents').all()).toEqual([{ deployment_id: job.deploymentId }]);
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
closeDb(); vi.resetModules();
const backend = await import('../../src/lib/server/backends/native.js');
expect(backend.nativeDeployment(job.deploymentId)).toBe(true);
await advance();
const original = await operationFor(job.deploymentId);
expect(original).toMatchObject({ kind: 'instance.create', payload: { name: job.name, port: 28960 } });
await advance();
expect((await operationFor(job.deploymentId))?.id).toBe(original!.id);
const restartedDb = (await import('../../src/lib/server/db/index.js')).db();
expect(restartedDb.prepare("SELECT id FROM ec_host_operations WHERE kind='instance.create'").all()).toHaveLength(1);
expect(restartedDb.prepare('SELECT protocol FROM ec_host_ports ORDER BY protocol').all()).toEqual([{ protocol: 'tcp' }, { protocol: 'udp' }]);
});
it('still rejects missing keys and stale lobby identities before creating an intent', async () => {
await prepareLibrary();
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
const input = { game: 't5' as const, gameMode: 'zombie' as const, mapSlug: 'theater', hostId, libraryId, image };
expect(queueNativeCreation(input)).toMatchObject({ ok: false, message: expect.stringContaining('server key') });
expect(queueNativeCreation({ ...input, plutoKey: 'saved-key', expectedName: 't5_zombie_theater_2' }))
.toMatchObject({ ok: false });
const { db } = await import('../../src/lib/server/db/index.js');
expect(db().prepare('SELECT deployment_id FROM native_creation_intents').all()).toEqual([]);
expect(db().prepare('SELECT id FROM server_deployments').all()).toEqual([]);
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
});
it('accepts cancellation through the provisioning API and rejects an already completed creation', async () => { it('accepts cancellation through the provisioning API and rejects an already completed creation', async () => {
const job = await queue(); const job = await queue();
const { POST } = await import('../../src/routes/api/provisioning/[id]/cancel/+server.js'); const { POST } = await import('../../src/routes/api/provisioning/[id]/cancel/+server.js');
@@ -341,11 +380,30 @@ describe('native instance provisioning', () => {
expect(db().prepare('SELECT COUNT(*) AS count FROM ec_host_ports').get()).toEqual({ count: 0 }); expect(db().prepare('SELECT COUNT(*) AS count FROM ec_host_ports').get()).toEqual({ count: 0 });
}); });
it('keeps a queued server pinned to its accepted files when newer game files arrive', async () => { it.each(['pinned', 'automatic'] as const)('keeps an unplaced %s launch pinned to its accepted files during cleanup', async placement => {
const job = await queue(); const job = await queue('t5', placement);
const acceptedId = libraryId; const acceptedId = libraryId;
const latest = await prepareLibrary('t5'); const latest = await prepareLibrary('t5');
expect(latest.id).not.toBe(acceptedId); expect(latest.id).not.toBe(acceptedId);
const { db } = await import('../../src/lib/server/db/index.js');
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
const { reconcileHostLibraryCleanup, cleanupControllerLibraries, getLibraryVersion, updateHostLibraryCaches }
= await import('../../src/lib/server/game-files/index.js');
const accepted = getLibraryVersion(acceptedId)!;
heartbeat(hostId, { ...inventory(), capabilities: ['library.remove'] });
expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]);
reconcileHostLibraryCleanup(hostId);
expect(db().prepare("SELECT id FROM ec_host_operations WHERE kind='library.remove' AND json_extract(payload,'$.libraryVersionId')=?")
.all(acceptedId)).toEqual([]);
// The unplaced intent also protects the controller's copy when every
// host reports it absent; its accepted version remains downloadable.
updateHostLibraryCaches(hostId, [{ libraryVersionId: acceptedId, status: 'missing' }]);
await cleanupControllerLibraries();
expect(db().prepare('SELECT version_id FROM game_file_cleanup WHERE version_id=?').get(acceptedId)).toBeUndefined();
const { stat } = await import('node:fs/promises');
expect((await stat(join(fixture.root, 'libraries', 't5', acceptedId, 'zone/base.ff'))).isFile()).toBe(true);
updateHostLibraryCaches(hostId, [{ libraryVersionId: acceptedId, status: 'ready', manifestSha256: accepted.manifestSha256 }]);
await advance(); await advance();
expect((await operationFor(job.deploymentId))?.payload.libraryId).toBe(acceptedId); expect((await operationFor(job.deploymentId))?.payload.libraryId).toBe(acceptedId);
const { nativeInstance } = await import('../../src/lib/server/backends/native.js'); const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
@@ -355,7 +413,8 @@ describe('native instance provisioning', () => {
it.each(['t5', 't6'] as const)('queues and verifies a %s deployment without Komodo', async game => { it.each(['t5', 't6'] as const)('queues and verifies a %s deployment without Komodo', async game => {
const job = await queue(game); await advance(); const job = await queue(game); await advance();
const operation = await operationFor(job.deploymentId); const operation = await operationFor(job.deploymentId);
expect(operation).toMatchObject({ kind: 'instance.create', hostId, generation: 1, state: 'pending', payload: { game, libraryId, image, port: job.port } }); expect(job.port).toBeNull();
expect(operation).toMatchObject({ kind: 'instance.create', hostId, generation: 1, state: 'pending', payload: { game, libraryId, image, port: 28960 } });
expect(operation?.payload.files).toEqual(expect.arrayContaining([expect.objectContaining({ path: 'configs/electric_cherry_map.cfg' })])); expect(operation?.payload.files).toEqual(expect.arrayContaining([expect.objectContaining({ path: 'configs/electric_cherry_map.cfg' })]));
await acknowledge(job.deploymentId); await acknowledge(job.deploymentId);
expect((await detail(job.deploymentId)).state).toBe('done'); expect((await detail(job.deploymentId)).state).toBe('done');
@@ -534,17 +593,22 @@ describe('native instance provisioning', () => {
const { getServerScript } = await import('../../src/lib/server/fleet/inventory.js'); const { getServerScript } = await import('../../src/lib/server/fleet/inventory.js');
expect(getServerScript(job.name, 'hud.gsc')?.sha256).toBe(createHash('sha256').update(after).digest('hex')); expect(getServerScript(job.name, 'hud.gsc')?.sha256).toBe(createHash('sha256').update(after).digest('hex'));
}); });
it('atomically launches mixed-game collections with per-host port reservations', async () => { it('atomically queues mixed-game collections and reserves per-host ports when dispatched', async () => {
const t5 = await prepareLibrary('t5'); const t6 = await prepareLibrary('t6'); const t5 = await prepareLibrary('t5'); const t6 = await prepareLibrary('t6');
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js'); const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
const blueprint = saveBlueprint({ name: 'Both games', entries: [{ game: 't5', mapSlug: 'theater', count: 2 }, { game: 't6', mapSlug: 'town', count: 1 }] }); const blueprint = saveBlueprint({ name: 'Both games', entries: [{ game: 't5', mapSlug: 'theater', count: 2 }, { game: 't6', mapSlug: 'town', count: 1 }] });
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js'); const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
const preview = previewCollection(blueprint.id, 'fill', { hostId, libraryIds: { t5: t5.id, t6: t6.id }, image }); const preview = previewCollection(blueprint.id, 'fill', { hostId, libraryIds: { t5: t5.id, t6: t6.id }, image });
expect(preview.ready).toBe(true); expect(preview.ready).toBe(true);
expect(preview.servers.map(server => server.port)).toEqual([28960, 28961, 28962]); expect(preview.servers.map(server => server.port)).toEqual([null, null, null]);
const queued = queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name])) }); const queued = queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name])) });
expect(queueCollection({ previewId: preview.id }).collectionId).toBe(queued.collectionId); expect(queueCollection({ previewId: preview.id }).collectionId).toBe(queued.collectionId);
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]);
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
saveFleetSettings({ ...fleetSettings(), createConcurrency: 4 });
await advance();
const instances = db().prepare('SELECT host_id,library_id,port FROM native_instances ORDER BY port').all(); const instances = db().prepare('SELECT host_id,library_id,port FROM native_instances ORDER BY port').all();
expect(instances).toEqual([{ host_id: hostId, library_id: t5.id, port: 28960 }, { host_id: hostId, library_id: t5.id, port: 28961 }, { host_id: hostId, library_id: t6.id, port: 28962 }]); expect(instances).toEqual([{ host_id: hostId, library_id: t5.id, port: 28960 }, { host_id: hostId, library_id: t5.id, port: 28961 }, { host_id: hostId, library_id: t6.id, port: 28962 }]);
const { createHost, createEnrollment, enrollHost, hostPortAvailable } = await import('../../src/lib/server/hosts/index.js'); const { createHost, createEnrollment, enrollHost, hostPortAvailable } = await import('../../src/lib/server/hosts/index.js');
@@ -554,13 +618,15 @@ describe('native instance provisioning', () => {
const { queueCreateServer } = await import('../../src/lib/server/provisioning/instances.js'); const { queueCreateServer } = await import('../../src/lib/server/provisioning/instances.js');
const second = queueCreateServer({ game: 't5', gameMode: 'zombie', mapSlug: 'moon', hostId: secondHost.id, libraryId: t5.id, image, plutoKey: 'second-host-key' }); const second = queueCreateServer({ game: 't5', gameMode: 'zombie', mapSlug: 'moon', hostId: secondHost.id, libraryId: t5.id, image, plutoKey: 'second-host-key' });
expect(second.ok).toBe(true); if (!second.ok) throw new Error(second.message); expect(second.ok).toBe(true); if (!second.ok) throw new Error(second.message);
expect(second.port).toBe(28960); expect(second.port).toBeNull();
await advance();
expect((await operationFor(second.deploymentId))?.payload.port).toBe(28960);
expect(hostPortAvailable(hostId, 28960)).toBe(false); expect(hostPortAvailable(hostId, 28960)).toBe(false);
expect(hostPortAvailable(secondHost.id, 28960)).toBe(false); expect(hostPortAvailable(secondHost.id, 28960)).toBe(false);
expect(db().prepare('SELECT COUNT(*) AS count FROM ec_host_ports').get()).toEqual({ count: 8 }); expect(db().prepare('SELECT COUNT(*) AS count FROM ec_host_ports').get()).toEqual({ count: 8 });
}); });
it('reads remote instance files through the agent including missing files and epoch timestamps', async () => { it('reads remote instance files through the agent including missing files and epoch timestamps', async () => {
const job = await queue(); const job = await queue(); await advance();
const backend = await import('../../src/lib/server/backends/native.js'); const backend = await import('../../src/lib/server/backends/native.js');
const remoteRead = vi.mocked(backend.nativeRequest); const remoteRead = vi.mocked(backend.nativeRequest);
remoteRead.mockImplementation(async (_instance, kind, payload) => { remoteRead.mockImplementation(async (_instance, kind, payload) => {
@@ -622,17 +688,17 @@ describe('native instance provisioning', () => {
const { getServerScript } = await import('../../src/lib/server/fleet/inventory.js'); const { getServerScript } = await import('../../src/lib/server/fleet/inventory.js');
expect(getServerScript(job.name, 'hud.gsc')).toBeUndefined(); expect(getServerScript(job.name, 'hud.gsc')).toBeUndefined();
}); });
it('cancels a queued native launch locally and releases both ports without submitting a create', async () => { it('cancels an unplaced native launch locally without reserving ports or submitting a create', async () => {
const job = await queue(); const job = await queue();
const { nativeInstance } = await import('../../src/lib/server/backends/native.js'); const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
const instance = nativeInstance(job.name)!; expect(nativeInstance(job.name)).toBeNull();
const { destroyServer } = await import('../../src/lib/server/provisioning/instances.js'); const { destroyServer } = await import('../../src/lib/server/provisioning/instances.js');
const cancelled = destroyServer(job.name); expect(cancelled.ok).toBe(true); const cancelled = destroyServer(job.name); expect(cancelled.ok).toBe(true);
await advance(); await advance();
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
expect((await detail(job.deploymentId)).state).toBe('cancelled'); expect((await detail(job.deploymentId)).state).toBe('cancelled');
expect(nativeInstance(job.name)).toBeNull(); expect(nativeInstance(job.name)).toBeNull();
expect(db().prepare('SELECT * FROM ec_host_ports WHERE instance_id=?').all(instance.id)).toEqual([]); expect(db().prepare('SELECT * FROM ec_host_ports').all()).toEqual([]);
expect(db().prepare("SELECT COUNT(*) AS count FROM ec_host_operations WHERE kind LIKE 'instance.%'").get()).toEqual({ count: 0 }); expect(db().prepare("SELECT COUNT(*) AS count FROM ec_host_operations WHERE kind LIKE 'instance.%'").get()).toEqual({ count: 0 });
const { getLibraryVersion } = await import('../../src/lib/server/game-files/index.js'); const { getLibraryVersion } = await import('../../src/lib/server/game-files/index.js');
expect(getLibraryVersion(libraryId)?.state).toBe('ready'); expect(getLibraryVersion(libraryId)?.state).toBe('ready');
@@ -661,7 +727,8 @@ describe('automatic capacity admission and placement', () => {
const { recipeForDeployment } = await import('../../src/lib/server/fleet/launch-recipes.js'); const { recipeForDeployment } = await import('../../src/lib/server/fleet/launch-recipes.js');
const result = queueCreateServer(input); const result = queueCreateServer(input);
expect(result.ok).toBe(true); if (!result.ok) return; expect(result.ok).toBe(true); if (!result.ok) return;
expect(nativeInstance(result.name)?.host_id).toBe(hostId); // Fast first fit. expect(nativeInstance(result.name)).toBeNull();
expect(result.port).toBeNull();
await advance(); await advance();
expect((await operationFor(result.deploymentId))?.hostId).toBe(other.id); expect((await operationFor(result.deploymentId))?.hostId).toBe(other.id);
expect(nativeInstance(result.name)).toMatchObject({ host_id: other.id, public_address: 'second.example' }); expect(nativeInstance(result.name)).toMatchObject({ host_id: other.id, public_address: 'second.example' });
@@ -689,13 +756,18 @@ describe('automatic capacity admission and placement', () => {
it.each([ it.each([
{ cpuUsagePercent: undefined }, { memoryAvailable: undefined }, { cpuCount: 0 }, { cpuUsagePercent: undefined }, { memoryAvailable: undefined }, { cpuCount: 0 },
{ cpuUsagePercent: 99 }, { memoryAvailable: 1024 ** 3 } { cpuUsagePercent: 99 }, { memoryAvailable: 1024 ** 3 }
])('rejects admission with unavailable resources: %j', async overrides => { ])('accepts an unplaced job and waits for unavailable resources: %j', async overrides => {
await prepareLibrary(); await prepareLibrary();
const { heartbeat } = await import('../../src/lib/server/hosts/index.js'); const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
heartbeat(hostId, { ...inventory(), ...overrides }); heartbeat(hostId, { ...inventory(), ...overrides });
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js'); const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
expect(queueNativeCreation(input)).toMatchObject({ ok: false, kind: 'conflict', message: expect.stringContaining('capacity') }); const queued = queueNativeCreation(input);
expect(queued).toMatchObject({ ok: true, port: null });
if (!queued.ok) throw new Error(queued.message);
await advance();
expect(await operationFor(queued.deploymentId)).toBeNull();
expect((await detail(queued.deploymentId)).stage).toContain('capacity');
expect(db().prepare('SELECT * FROM native_instances').all()).toEqual([]); expect(db().prepare('SELECT * FROM native_instances').all()).toEqual([]);
expect(db().prepare('SELECT * FROM ec_host_ports').all()).toEqual([]); expect(db().prepare('SELECT * FROM ec_host_ports').all()).toEqual([]);
}); });
@@ -709,22 +781,28 @@ describe('automatic capacity admission and placement', () => {
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
db().prepare("UPDATE ec_hosts SET last_heartbeat='2000-01-01' WHERE id=?").run(hostId); db().prepare("UPDATE ec_hosts SET last_heartbeat='2000-01-01' WHERE id=?").run(hostId);
const result = queueNativeCreation({ ...input, previewId: preview.id }); const result = queueNativeCreation({ ...input, previewId: preview.id });
expect(result).toMatchObject({ ok: true, name: preview.name, port: 30000 }); expect(result).toMatchObject({ ok: true, name: preview.name, port: null });
expect(queueNativeCreation({ ...input, previewId: preview.id })).toEqual(result); expect(queueNativeCreation({ ...input, previewId: preview.id })).toEqual(result);
if (!result.ok) throw new Error(result.message);
await advance();
expect((await operationFor(result.deploymentId))?.payload.port).toBe(30000);
}); });
it('rejects an entire collection when capacity shrinks after its preview', async () => { it('queues the whole collection and waits when capacity shrinks after its preview', async () => {
await prepareLibrary(); await prepareLibrary();
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js'); const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js'); const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
const blueprint = saveBlueprint({ name: 'Two lobbies', entries: [{ game: 't5', mapSlug: 'theater', count: 2 }] }); const blueprint = saveBlueprint({ name: 'Two lobbies', entries: [{ game: 't5', mapSlug: 'theater', count: 2 }] });
const preview = previewCollection(blueprint.id); const preview = previewCollection(blueprint.id);
const { heartbeat } = await import('../../src/lib/server/hosts/index.js'); const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
heartbeat(hostId, { ...inventory(), cpuCount: 1, cpuUsagePercent: 0 }); heartbeat(hostId, { ...inventory(), cpuUsagePercent: 100 });
expect(() => queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key'])) })).toThrow('capacity'); const result = queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key'])) });
expect(result).toMatchObject({ ok: true, collectionId: expect.any(String) });
await advance();
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
expect(db().prepare('SELECT * FROM native_instances').all()).toEqual([]); expect(db().prepare('SELECT * FROM native_instances').all()).toEqual([]);
expect(db().prepare('SELECT * FROM ec_host_ports').all()).toEqual([]); expect(db().prepare('SELECT * FROM ec_host_ports').all()).toEqual([]);
expect(db().prepare('SELECT * FROM server_launch_collections').all()).toEqual([]); expect(db().prepare('SELECT * FROM server_launch_collections').all()).toHaveLength(1);
expect(db().prepare('SELECT * FROM native_creation_intents').all()).toHaveLength(2);
}); });
it.each([ it.each([
{ resource: 'CPU', measurements: { cpuCount: 4, cpuUsagePercent: 82.5 }, over: { cpuUsagePercent: 82.51 } }, { resource: 'CPU', measurements: { cpuCount: 4, cpuUsagePercent: 82.5 }, over: { cpuUsagePercent: 82.51 } },
@@ -742,15 +820,19 @@ describe('automatic capacity admission and placement', () => {
expect(() => choosePlacement(options)).toThrow('capacity'); expect(() => choosePlacement(options)).toThrow('capacity');
expect(() => choosePlacement({ ...options, thorough: true })).toThrow('capacity'); expect(() => choosePlacement({ ...options, thorough: true })).toThrow('capacity');
}); });
it('uses calibrated footprints to admit more instances than the cold-start allowance', async () => { it('dispatches a waiting instance after calibration lowers the cold-start allowance', async () => {
const first = await queue(); const first = await queue();
await advance(); await acknowledge(first.deploymentId);
const { heartbeat } = await import('../../src/lib/server/hosts/index.js'); const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
const { nativeInstance } = await import('../../src/lib/server/backends/native.js'); const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
const { recordInstanceResources, capacityCalibration } = await import('../../src/lib/server/hosts/calibration.js'); const { recordInstanceResources, capacityCalibration } = await import('../../src/lib/server/hosts/calibration.js');
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js'); const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
const measured = { ...inventory(), cpuCount: 1, cpuUsagePercent: 0 }; const measured = { ...inventory(), cpuCount: 1, cpuUsagePercent: 50 };
heartbeat(hostId, measured); heartbeat(hostId, measured);
expect(queueNativeCreation(input)).toMatchObject({ ok: false, kind: 'conflict' }); const next = queueNativeCreation(input);
expect(next.ok).toBe(true); if (!next.ok) throw new Error(next.message);
await advance();
expect(await operationFor(next.deploymentId)).toBeNull();
const instance = nativeInstance(first.name)!; const instance = nativeInstance(first.name)!;
for (let minute = 0; minute < 25; minute++) { for (let minute = 0; minute < 25; minute++) {
recordInstanceResources(hostId, { ...measured, instanceResources: [{ instanceId: instance.id, generation: 1, recordInstanceResources(hostId, { ...measured, instanceResources: [{ instanceId: instance.id, generation: 1,
@@ -758,7 +840,8 @@ describe('automatic capacity admission and placement', () => {
new Date(Date.now() - (25 - minute) * 60_000).toISOString()); new Date(Date.now() - (25 - minute) * 60_000).toISOString());
} }
expect(capacityCalibration(hostId).startup.source).toBe('host'); expect(capacityCalibration(hostId).startup.source).toBe('host');
expect(queueNativeCreation(input)).toMatchObject({ ok: true }); await advance();
expect(await operationFor(next.deploymentId)).not.toBeNull();
}); });
it('queues a collection at running demand and releases startup reservations as live usage arrives', async () => { it('queues a collection at running demand and releases startup reservations as live usage arrives', async () => {
await prepareLibrary(); await prepareLibrary();
@@ -791,7 +874,7 @@ describe('automatic capacity admission and placement', () => {
await advance(); await advance();
expect(creates()).toHaveLength(2); // Its actual usage replaces the peak even before health verification. expect(creates()).toHaveLength(2); // Its actual usage replaces the peak even before health verification.
}); });
it('charges queued reservations and releases them on cancellation', async () => { it('reserves queued lobby identities without host resources and releases them on cancellation', async () => {
await prepareLibrary(); await prepareLibrary();
const { heartbeat } = await import('../../src/lib/server/hosts/index.js'); const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
heartbeat(hostId, { ...inventory(), cpuCount: 1, cpuUsagePercent: 0 }); heartbeat(hostId, { ...inventory(), cpuCount: 1, cpuUsagePercent: 0 });
@@ -799,9 +882,13 @@ describe('automatic capacity admission and placement', () => {
const { cancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js'); const { cancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
const first = queueNativeCreation(input); const first = queueNativeCreation(input);
expect(first.ok).toBe(true); if (!first.ok) return; expect(first.ok).toBe(true); if (!first.ok) return;
expect(queueNativeCreation(input)).toMatchObject({ ok: false, kind: 'conflict' }); const second = queueNativeCreation(input);
expect(second).toMatchObject({ ok: true, name: 't5_zombie_theater_2', port: null });
const { db } = await import('../../src/lib/server/db/index.js');
expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]);
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
cancelCreation(first.deploymentId); cancelCreation(first.deploymentId);
expect(queueNativeCreation(input)).toMatchObject({ ok: true }); expect(queueNativeCreation(input)).toMatchObject({ ok: true, name: first.name, port: null });
}); });
it.each(['memory', 'cpu'] as const)('dispatches immediately after %s recovers instead of waiting for five-minute history', async resource => { it.each(['memory', 'cpu'] as const)('dispatches immediately after %s recovers instead of waiting for five-minute history', async resource => {
await prepareLibrary(); await prepareLibrary();
@@ -846,7 +933,10 @@ describe('automatic capacity admission and placement', () => {
// even though the next host heartbeat has not observed it yet. // even though the next host heartbeat has not observed it yet.
await advance(); await advance();
expect(db().prepare("SELECT * FROM ec_host_operations WHERE kind='instance.create'").all()).toHaveLength(1); expect(db().prepare("SELECT * FROM ec_host_operations WHERE kind='instance.create'").all()).toHaveLength(1);
expect(queueNativeCreation(input)).toMatchObject({ ok: false, kind: 'conflict' }); const later = queueNativeCreation(input);
expect(later.ok).toBe(true); if (!later.ok) throw new Error(later.message);
await advance();
expect(await operationFor(later.deploymentId)).toBeNull();
}); });
it('does not charge a starting container twice when its usage is already in the heartbeat', async () => { it('does not charge a starting container twice when its usage is already in the heartbeat', async () => {
await prepareLibrary(); await prepareLibrary();
@@ -873,6 +963,8 @@ describe('automatic capacity admission and placement', () => {
}); });
it.each(['missing', 'old-generation'] as const)('keeps full reservations for a submitted launch with %s measurements', async evidence => { it.each(['missing', 'old-generation'] as const)('keeps full reservations for a submitted launch with %s measurements', async evidence => {
await prepareLibrary(); await prepareLibrary();
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
saveFleetSettings({ ...fleetSettings(), createConcurrency: 2 });
const { heartbeat } = await import('../../src/lib/server/hosts/index.js'); const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js'); const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
const { nativeInstance } = await import('../../src/lib/server/backends/native.js'); const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
@@ -885,8 +977,11 @@ describe('automatic capacity admission and placement', () => {
heartbeat(hostId, { ...inventory(), memoryTotal: 3 * GiB, memoryAvailable: 2 * GiB, heartbeat(hostId, { ...inventory(), memoryTotal: 3 * GiB, memoryAvailable: 2 * GiB,
instanceResources: evidence === 'missing' ? [] : [{ instanceId: instance.id, generation: instance.generation + 1, instanceResources: evidence === 'missing' ? [] : [{ instanceId: instance.id, generation: instance.generation + 1,
uptimeSeconds: 60, cpuCores: 0.1, memoryBytes: Math.floor(0.8 * GiB) }] }); uptimeSeconds: 60, cpuCores: 0.1, memoryBytes: Math.floor(0.8 * GiB) }] });
expect(queueNativeCreation(input)).toMatchObject({ ok: false, kind: 'conflict', const next = queueNativeCreation(input);
message: expect.stringContaining('3.00 GiB projected, 2.53 GiB usable') }); expect(next.ok).toBe(true); if (!next.ok) throw new Error(next.message);
await advance();
expect(await operationFor(next.deploymentId)).toBeNull();
expect((await detail(next.deploymentId)).stage).toContain('3.00 GiB projected, 2.53 GiB usable');
}); });
it('waits when capacity disappears after admission without sending a create command', async () => { it('waits when capacity disappears after admission without sending a create command', async () => {
await prepareLibrary(); await prepareLibrary();
@@ -931,11 +1026,16 @@ describe('automatic capacity admission and placement', () => {
const { nativeInstance } = await import('../../src/lib/server/backends/native.js'); const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
const result = queueNativeCreation(input); const result = queueNativeCreation(input);
expect(result.ok).toBe(true); if (!result.ok) return; expect(result.ok).toBe(true); if (!result.ok) return;
expect(nativeInstance(result.name)).toBeNull();
await advance();
expect(nativeInstance(result.name)?.host_id).toBe(other.id); expect(nativeInstance(result.name)?.host_id).toBe(other.id);
updateHost(other.id, { mode: 'draining' }); updateHost(other.id, { mode: 'draining' });
expect(queueNativeCreation(input)).toMatchObject({ ok: false, kind: 'conflict' }); const next = queueNativeCreation(input);
expect(next.ok).toBe(true); if (!next.ok) throw new Error(next.message);
await advance();
expect(await operationFor(next.deploymentId)).toBeNull();
}); });
it('spreads collection admission across hosts, allowing the same port on different hosts', async () => { it('spreads collection dispatch across hosts, allowing the same port on different hosts', async () => {
await prepareLibrary(); await secondHost({ cpuCount: 1, cpuUsagePercent: 0 }); await prepareLibrary(); await secondHost({ cpuCount: 1, cpuUsagePercent: 0 });
const { heartbeat } = await import('../../src/lib/server/hosts/index.js'); const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
heartbeat(hostId, { ...inventory(), cpuCount: 1, cpuUsagePercent: 0 }); heartbeat(hostId, { ...inventory(), cpuCount: 1, cpuUsagePercent: 0 });
@@ -944,10 +1044,17 @@ describe('automatic capacity admission and placement', () => {
const blueprint = saveBlueprint({ name: 'Two lobbies', entries: [{ game: 't5', mapSlug: 'theater', count: 2 }] }); const blueprint = saveBlueprint({ name: 'Two lobbies', entries: [{ game: 't5', mapSlug: 'theater', count: 2 }] });
const preview = previewCollection(blueprint.id); const preview = previewCollection(blueprint.id);
expect(preview.ready).toBe(true); expect(preview.ready).toBe(true);
expect(preview.servers.map(server => server.port)).toEqual([28960, 28960]); expect(preview.servers.map(server => server.port)).toEqual([null, null]);
expect(new Set(preview.servers.map(server => server.native?.hostId)).size).toBe(2);
expect(queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key'])) }).ok).toBe(true); expect(queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key'])) }).ok).toBe(true);
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]);
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
saveFleetSettings({ ...fleetSettings(), createConcurrency: 2 });
await advance();
// Concurrent ranking can select the same host before the first
// reservation commits. The held job retries against the new usage.
await advance();
expect(db().prepare('SELECT DISTINCT host_id FROM native_instances').all()).toHaveLength(2); expect(db().prepare('SELECT DISTINCT host_id FROM native_instances').all()).toHaveLength(2);
expect(db().prepare('SELECT port FROM native_instances').all()).toEqual([{ port: 28960 }, { port: 28960 }]);
}); });
}); });