Queue native collection launches before async placement
This commit is contained in:
@@ -39,10 +39,11 @@ including SQLite metadata and published game-file storage.
|
|||||||
shows setup progress. Automatic agent updates are enabled by default.
|
shows setup progress. Automatic agent updates are enabled by default.
|
||||||
3. Add the exact Plutonium server keys in **Keys**, or supply missing keys when
|
3. Add the exact Plutonium server keys in **Keys**, or supply missing keys when
|
||||||
launching.
|
launching.
|
||||||
4. Launch a server or collection. EC quickly checks CPU, memory and port capacity,
|
4. Launch a server or collection. Collection previews show cached capacity
|
||||||
then automatically places each server on the least loaded eligible host. The
|
estimates; Apply saves the queue and opens its progress page. Provisioning
|
||||||
latest game files and default runtime are selected automatically. EC reserves
|
checks capacity and assigns each server a host and port when it can start.
|
||||||
resources for queued launches and verifies startup. CPU/memory allowances learn
|
The latest game files and default runtime are selected automatically.
|
||||||
|
CPU/memory allowances learn
|
||||||
from actual container usage, using temporary defaults while calibrating.
|
from actual container usage, using temporary defaults while calibrating.
|
||||||
|
|
||||||
See [Native host management](docs/native-hosts.md) for installation, credentials,
|
See [Native host management](docs/native-hosts.md) for installation, credentials,
|
||||||
|
|||||||
@@ -29,7 +29,7 @@ It fills deficits without changing existing servers' files.
|
|||||||
2. Add maps from Black Ops I or II and set the desired total lobbies for each.
|
2. Add maps from Black Ops I or II and set the desired total lobbies for each.
|
||||||
3. Choose **Save & preview** (or **Preview fleet changes** on a saved collection).
|
3. Choose **Save & preview** (or **Preview fleet changes** on a saved collection).
|
||||||
Choose **Fill missing servers** or **Match collection exactly**, then review
|
Choose **Fill missing servers** or **Match collection exactly**, then review
|
||||||
the server IDs and game ports.
|
the server IDs and estimated capacity. Native host ports are assigned later.
|
||||||
4. Lobbies with a saved key show **Saved key ready** and reuse it automatically.
|
4. Lobbies with a saved key show **Saved key ready** and reuse it automatically.
|
||||||
Enter a Plutonium key only for lobbies missing one. Update saved keys in **Keys**.
|
Enter a Plutonium key only for lobbies missing one. Update saved keys in **Keys**.
|
||||||
5. Choose **Queue N servers** (or confirm removals and choose **Apply** in exact
|
5. Choose **Queue N servers** (or confirm removals and choose **Apply** in exact
|
||||||
@@ -51,17 +51,18 @@ Unstarted creations are cancelled without booting a container. Running servers
|
|||||||
use the normal removal queue and require a fresh zero-player RCON response.
|
use the normal removal queue and require a fresh zero-player RCON response.
|
||||||
Occupied or unreachable servers wait. Already pending removals are reused.
|
Occupied or unreachable servers wait. Already pending removals are reused.
|
||||||
|
|
||||||
Creation reservations and removals are queued in one database transaction. A
|
Creation intents and removals are queued in one database transaction. A
|
||||||
missing key or changed fleet rejects the entire request. Any fleet identity,
|
missing key or changed fleet rejects the entire request. Any fleet identity,
|
||||||
map, port, or pending-removal change since an exact preview requires a new review.
|
map, port, or pending-removal change since an exact preview requires a new review.
|
||||||
Servers added after submission are not included. Applying the same preview again
|
Servers added after submission are not included. Applying the same preview again
|
||||||
returns the same collection without repeating operations.
|
returns the same collection without repeating operations.
|
||||||
|
|
||||||
Ports belonging to reviewed removals can be reserved for new lobbies even when
|
Native launches can be queued even when the current port range is full. Removed
|
||||||
the current port range is full. A new lobby waits until its port's removal has
|
servers retain their ports until cleanup completes; provisioning assigns newly
|
||||||
completed; a failed, interrupted, or cancelled removal keeps that launch blocked,
|
available ports as each creation starts. A failed or cancelled removal can leave
|
||||||
with the reason shown in its progress. Other launches with available capacity can
|
creations waiting for capacity, while other eligible hosts can still be used.
|
||||||
proceed. Reservations and dependencies survive app restarts. The progress page
|
Legacy launches retain their reviewed port dependencies. Queued work and any
|
||||||
|
assigned reservations survive app restarts. The progress page
|
||||||
tracks creations and removals, including removal-only plans, and lists the servers
|
tracks creations and removals, including removal-only plans, and lists the servers
|
||||||
that were retained. The existing operation detail page offers removal cancellation
|
that were retained. The existing operation detail page offers removal cancellation
|
||||||
and cleanup recovery where applicable.
|
and cleanup recovery where applicable.
|
||||||
@@ -79,18 +80,25 @@ Collections saved by the earlier single-server release load as one-server lineup
|
|||||||
and can be expanded directly. Any file snapshots in older collections are ignored
|
and can be expanded directly. Any file snapshots in older collections are ignored
|
||||||
when preparing new launches.
|
when preparing new launches.
|
||||||
|
|
||||||
A collection preview does not reserve capacity. At submission, all lobby IDs,
|
A native collection preview reads a shared capacity snapshot, maintained from host
|
||||||
ports, and keys are validated again. The entire collection is then queued in one
|
heartbeats and fleet changes. It estimates available ports, CPU, memory, and startup
|
||||||
database transaction. If any server lacks a valid key or capacity has changed,
|
slots without running placement for each server. Missing or insufficient capacity
|
||||||
no jobs, reservations, or key changes from that submission are committed. Refreshing
|
is a warning: provisioning can wait for it. The preview does not reserve resources.
|
||||||
|
|
||||||
|
Apply validates the reviewed configuration, lobby IDs, keys, and fleet changes, saves all jobs in
|
||||||
|
one database transaction, and returns the progress page ID. Native jobs initially
|
||||||
|
have no host or port; each job performs its capacity checks asynchronously during
|
||||||
|
provisioning. If a server lacks a valid key or its reviewed identity changed,
|
||||||
|
no jobs or key changes from that submission are committed. Legacy submissions also
|
||||||
|
revalidate their reviewed ports. Refreshing
|
||||||
the preview retains typed keys only for unchanged server IDs still missing a saved
|
the preview retains typed keys only for unchanged server IDs still missing a saved
|
||||||
key, and resets removal confirmation.
|
key, and resets removal confirmation.
|
||||||
|
|
||||||
Submitting the same preview again returns the same collection, including after
|
Submitting the same preview again returns the same collection, including after
|
||||||
a lost response or an application restart. Reusing the collection with a new
|
a lost response or an application restart. Reusing the collection with a new
|
||||||
preview recomputes the shortfall, including servers queued by earlier launches.
|
preview recomputes the shortfall, including servers queued by earlier launches.
|
||||||
It does not add another full copy of the lineup. If the shortfall or assigned
|
It does not add another full copy of the lineup. If the shortfall or reviewed
|
||||||
capacity changes before submission, the request is rejected without changes and
|
identities change before submission, the request is rejected without changes and
|
||||||
requires a fresh preview. Previously saved additive previews are checked against
|
requires a fresh preview. Previously saved additive previews are checked against
|
||||||
the current shortfall too.
|
the current shortfall too.
|
||||||
|
|
||||||
@@ -101,7 +109,7 @@ Only then can the next queued server start. Set **Concurrent server startups** i
|
|||||||
a positive integer to allow more simultaneous startups. Each worker pass admits
|
a positive integer to allow more simultaneous startups. Each worker pass admits
|
||||||
at most the available slots, even if an attempt fails immediately.
|
at most the available slots, even if an attempt fails immediately.
|
||||||
|
|
||||||
Names, ports, and queue order survive application restarts. A server still booting
|
Names, any assigned ports, and queue order survive application restarts. A server still booting
|
||||||
or waiting for health keeps its slot until it passes verification or is removed.
|
or waiting for health keeps its slot until it passes verification or is removed.
|
||||||
A lost startup response also keeps the slot because the container may already be
|
A lost startup response also keeps the slot because the container may already be
|
||||||
running. A confirmed creation failure is recorded for review, allowing later
|
running. A confirmed creation failure is recorded for review, allowing later
|
||||||
@@ -125,7 +133,7 @@ remain visible in Provisioning activity and history.
|
|||||||
5. Choose **Launch server** and follow the operation to verified health.
|
5. Choose **Launch server** and follow the operation to verified health.
|
||||||
|
|
||||||
A preview lasts 30 minutes. It neither reserves capacity nor saves a key. If
|
A preview lasts 30 minutes. It neither reserves capacity nor saves a key. If
|
||||||
another launch takes the lobby or port, submission stops before saving the key;
|
another launch takes the lobby, submission stops before saving the key;
|
||||||
refresh the preview and review the newly assigned ID. Refreshing a preview clears
|
refresh the preview and review the newly assigned ID. Refreshing a preview clears
|
||||||
the typed key so it cannot accidentally be saved against a different lobby.
|
the typed key so it cannot accidentally be saved against a different lobby.
|
||||||
|
|
||||||
@@ -161,14 +169,15 @@ without a recorded recipe retain their history but cannot offer this action.
|
|||||||
|
|
||||||
A preview shows the currently promoted managed files, configuration, and image
|
A preview shows the currently promoted managed files, configuration, and image
|
||||||
reference for the selected map. It does not pin those versions. New launches
|
reference for the selected map. It does not pin those versions. New launches
|
||||||
resolve the promoted setup again when submitted and when provisioning begins, so
|
resolve the promoted setup again when provisioning begins, so
|
||||||
promotions made after a preview or while a job is queued apply to that server.
|
promotions made after a preview or while a job is queued apply to that server.
|
||||||
This applies to collection collections, single-server launches, and **Launch again**.
|
This applies to collections, single-server launches, and **Launch again**.
|
||||||
|
|
||||||
Previews and queued jobs store file metadata without copying script or plugin
|
Previews and queued jobs store file metadata without copying script or plugin
|
||||||
contents for every lobby. Provisioning captures the bytes when each job starts.
|
contents for every lobby. Provisioning captures the bytes when each job starts.
|
||||||
Native launch preparation reads files and encodes command payloads in a worker
|
Native host ranking, file reads, and command encoding run in a worker thread.
|
||||||
thread. The controller rechecks capacity, promotions, credentials, and safety
|
The controller makes a short atomic reservation check after ranking, then
|
||||||
|
rechecks capacity, promotions, credentials, and safety
|
||||||
before submitting those commands. Agent delivery streams the stored payloads in
|
before submitting those commands. Agent delivery streams the stored payloads in
|
||||||
small chunks, while progress polling reads metadata without loading file bytes.
|
small chunks, while progress polling reads metadata without loading file bytes.
|
||||||
|
|
||||||
@@ -180,7 +189,8 @@ also use the promoted setup at provisioning time.
|
|||||||
|
|
||||||
A submitted preview is bound to one deployment: retrying the same submission
|
A submitted preview is bound to one deployment: retrying the same submission
|
||||||
returns the original operation, including after an application restart or a lost
|
returns the original operation, including after an application restart or a lost
|
||||||
HTTP response. The reviewed game/map, lobby ID, port, and key checks are preserved.
|
HTTP response. The reviewed game/map, lobby ID, and key checks are preserved;
|
||||||
|
native ports are reserved only during provisioning.
|
||||||
Already running servers continue to receive updates through campaigns, betas,
|
Already running servers continue to receive updates through campaigns, betas,
|
||||||
and drift reconciliation using the server's game/map scopes.
|
and drift reconciliation using the server's game/map scopes.
|
||||||
Container tags can resolve to newer images; use a digest for a fixed image.
|
Container tags can resolve to newer images; use a digest for a fixed image.
|
||||||
|
|||||||
+22
-14
@@ -34,11 +34,12 @@ Status replies still expire normally, and native timeout diagnostics are retaine
|
|||||||
progress and errors. Wait for verification before launching. Hosts report
|
progress and errors. Wait for verification before launching. Hosts report
|
||||||
connection and readiness separately; deployment also checks the particular
|
connection and readiness separately; deployment also checks the particular
|
||||||
game files, image, capacity and ports required for the launch.
|
game files, image, capacity and ports required for the launch.
|
||||||
6. Launch a server or collection. The host, latest game files and default
|
6. Launch a server or collection. The latest game files and default runtime
|
||||||
runtime are selected automatically after checking capacity. An advanced runtime override is
|
are selected automatically. An advanced runtime override is
|
||||||
available. If the latest files change after previewing, refresh the preview
|
available. If the latest files change after previewing, refresh the preview
|
||||||
before launching.
|
before launching.
|
||||||
EC selects lobby IDs and reserves UDP/TCP ports atomically.
|
Apply saves the requested lobby IDs. Provisioning selects a host and atomically
|
||||||
|
reserves UDP/TCP ports when capacity is available.
|
||||||
Saved Plutonium server keys remain associated with their exact server names.
|
Saved Plutonium server keys remain associated with their exact server names.
|
||||||
|
|
||||||
A fresh destination host downloads centrally stored game files without manual
|
A fresh destination host downloads centrally stored game files without manual
|
||||||
@@ -133,14 +134,19 @@ remain unknown). Scheduling checks capacity against the latest heartbeat and use
|
|||||||
the last five minutes to rank eligible hosts; the Inventory panel shows the latest
|
the last five minutes to rank eligible hosts; the Inventory panel shows the latest
|
||||||
reading.
|
reading.
|
||||||
|
|
||||||
New servers are placed automatically. Preview and submission perform local
|
New servers are placed automatically. Collection previews use cached availability
|
||||||
capacity checks without contacting agents: a connected, active host must have
|
estimates and submission saves unplaced jobs without running capacity checks.
|
||||||
|
Provisioning checks each job asynchronously: a connected, active host must have
|
||||||
verified game files, its runtime, storage, a free UDP/TCP port, and valid CPU and
|
verified game files, its runtime, storage, a free UDP/TCP port, and valid CPU and
|
||||||
memory measurements. Hosts with stale or missing measurements cannot accept new
|
memory measurements. Hosts with stale or missing measurements cannot accept new
|
||||||
servers. Existing API clients may still explicitly constrain a launch to a host.
|
servers. Existing API clients may still explicitly constrain a launch to a host.
|
||||||
|
|
||||||
Admission reserves a provisional host and port atomically, including collection
|
Queued jobs retain their accepted game-file version. Automatic host preparation
|
||||||
launches. Agent **1.4.6** fixes container discovery for resource sampling (the
|
also downloads the files and runtime images needed by unplaced jobs; a newer
|
||||||
|
upload does not remove files still needed by the queue.
|
||||||
|
|
||||||
|
Only provisioning reserves a host and port. Agent **1.4.6** fixes container
|
||||||
|
discovery for resource sampling (the
|
||||||
earlier collector omitted `label=` from its Docker filter). It samples each
|
earlier collector omitted `label=` from its Docker filter). It samples each
|
||||||
managed running container with a bounded
|
managed running container with a bounded
|
||||||
[`docker stats`](https://docs.docker.com/reference/cli/docker/container/stats/)
|
[`docker stats`](https://docs.docker.com/reference/cli/docker/container/stats/)
|
||||||
@@ -180,19 +186,19 @@ heartbeat ingestion, at most once per minute per host/fleet phase; new peaks
|
|||||||
raise the cache immediately. Launch admission reads only the cached model.
|
raise the cache immediately. Launch admission reads only the cached model.
|
||||||
|
|
||||||
Scheduling still leaves **5% host CPU and 500 MB RAM (500,000,000 bytes)** for the OS and bursts.
|
Scheduling still leaves **5% host CPU and 500 MB RAM (500,000,000 bytes)** for the OS and bursts.
|
||||||
These allowances are scheduling estimates, not Docker limits. Collection admission
|
These allowances are scheduling estimates, not Docker limits. Collection previews
|
||||||
budgets new and never-submitted queued servers at their running allowance, rather
|
estimate how many new servers fit at their running allowance. Jobs may still be
|
||||||
than assuming the entire collection starts simultaneously. Current host CPU and
|
queued when that estimate is insufficient. Current host CPU and
|
||||||
available memory account for existing containers and other processes. Observed
|
available memory account for existing containers and other processes. Observed
|
||||||
containers are not topped up to historical startup or running peaks, even while
|
containers are not topped up to historical startup or running peaks, even while
|
||||||
awaiting health verification. Established instances are covered by host usage
|
awaiting health verification. Established instances are covered by host usage
|
||||||
when individual container measurements are unavailable.
|
when individual container measurements are unavailable.
|
||||||
|
|
||||||
Dispatch checks the next launch's startup allowance against live host usage.
|
Dispatch checks the next launch's startup allowance against live host usage.
|
||||||
Other never-submitted queued launches wait their turn and retain their port
|
Unplaced queued launches wait their turn without holding physical resources.
|
||||||
reservations. Submitted or uncertain launches missing a matching-generation
|
Any previously assigned reservations remain protected. Submitted or uncertain launches missing a matching-generation
|
||||||
measurement still consume a full startup allowance until telemetry accounts for
|
measurement still consume a full startup allowance until telemetry accounts for
|
||||||
them. This lets collections queue based on running demand and start as capacity
|
them. This lets collections start as capacity
|
||||||
becomes available, without stacking every queued server's startup peak.
|
becomes available, without stacking every queued server's startup peak.
|
||||||
Capacity errors report projected usage, usable capacity and reservations.
|
Capacity errors report projected usage, usable capacity and reservations.
|
||||||
|
|
||||||
@@ -201,7 +207,9 @@ current CPU and available memory against reservations and headroom.
|
|||||||
Among hosts that fit, it chooses the lowest projected CPU/memory utilization,
|
Among hosts that fit, it chooses the lowest projected CPU/memory utilization,
|
||||||
using the busier of the latest reading and the five-minute average for ranking.
|
using the busier of the latest reading and the five-minute average for ranking.
|
||||||
A historical resource dip cannot block a host whose current capacity has recovered.
|
A historical resource dip cannot block a host whose current capacity has recovered.
|
||||||
Provisioning moves the provisional reservation atomically and records the final host in the launch recipe. Host network defaults
|
Historical ranking runs in a worker thread. The controller rechecks current
|
||||||
|
capacity and reserves the selected host and port atomically, then records them
|
||||||
|
in the launch recipe. Host network defaults
|
||||||
are resolved again for the final host; explicit overrides are preserved. If no
|
are resolved again for the final host; explicit overrides are preserved. If no
|
||||||
host qualifies, the request waits and is checked again. Once remote staging or
|
host qualifies, the request waits and is checked again. Once remote staging or
|
||||||
creation has been submitted, placement stays fixed to prevent duplicate instances.
|
creation has been submitted, placement stays fixed to prevent duplicate instances.
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ export interface LaunchFileSummary {
|
|||||||
}
|
}
|
||||||
|
|
||||||
export interface LaunchRecipeSummary {
|
export interface LaunchRecipeSummary {
|
||||||
native?: { hostId: string; libraryId: string; image?: string; bindAddress?: string; publicAddress?: string; publicPort?: number };
|
native?: import('./hosts.js').NativePlacement;
|
||||||
game: ZombieGame;
|
game: ZombieGame;
|
||||||
mapSlug: string;
|
mapSlug: string;
|
||||||
mapName: string;
|
mapName: string;
|
||||||
@@ -35,6 +35,18 @@ export interface BlueprintSummary {
|
|||||||
|
|
||||||
export type CollectionMode = 'fill' | 'exact';
|
export type CollectionMode = 'fill' | 'exact';
|
||||||
|
|
||||||
|
export interface LaunchAvailability {
|
||||||
|
updatedAt: string | null;
|
||||||
|
requested: number;
|
||||||
|
estimatedAvailable: number | null;
|
||||||
|
portsAvailable: number | null;
|
||||||
|
cpuCoresAvailable: number | null;
|
||||||
|
memoryBytesAvailable: number | null;
|
||||||
|
startupSlots: number | null;
|
||||||
|
startupLimit: number | null;
|
||||||
|
queued: number | null;
|
||||||
|
}
|
||||||
|
|
||||||
export interface CollectionPreview {
|
export interface CollectionPreview {
|
||||||
id: string;
|
id: string;
|
||||||
expiresAt: string;
|
expiresAt: string;
|
||||||
@@ -47,6 +59,8 @@ export interface CollectionPreview {
|
|||||||
coverage: { name: string; health: string; href: string }[]; recipe: LaunchRecipeSummary }[];
|
coverage: { name: string; health: string; href: string }[]; recipe: LaunchRecipeSummary }[];
|
||||||
servers: { native?: import('./hosts.js').NativePlacement; entryIndex: number; name: string | null; port: number | null; hasSavedKey: boolean }[];
|
servers: { native?: import('./hosts.js').NativePlacement; entryIndex: number; name: string | null; port: number | null; hasSavedKey: boolean }[];
|
||||||
blockers: string[];
|
blockers: string[];
|
||||||
|
warnings?: string[];
|
||||||
|
availability?: LaunchAvailability;
|
||||||
ready: boolean;
|
ready: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+2
-2
@@ -116,9 +116,9 @@ export interface NativeDeploymentSelection {
|
|||||||
bindAddress?: string; publicAddress?: string; publicPort?: number;
|
bindAddress?: string; publicAddress?: string; publicPort?: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** The host/port are provisional until the creation command is submitted. */
|
/** Queued automatic launches have no assigned host or port until provisioning. */
|
||||||
export interface NativePlacement {
|
export interface NativePlacement {
|
||||||
hostId: string; libraryId: string; image?: string;
|
hostId?: string; libraryId: string; image?: string;
|
||||||
bindAddress?: string; publicAddress?: string; publicPort?: number;
|
bindAddress?: string; publicAddress?: string; publicPort?: number;
|
||||||
automatic?: boolean;
|
automatic?: boolean;
|
||||||
network?: { bindAddress?: string; publicAddress?: string; publicPort?: number };
|
network?: { bindAddress?: string; publicAddress?: string; publicPort?: number };
|
||||||
|
|||||||
@@ -15,7 +15,19 @@ export function nativeInstance(server: string): NativeInstance | null {
|
|||||||
}
|
}
|
||||||
|
|
||||||
export function nativeDeployment(id: string): boolean {
|
export function nativeDeployment(id: string): boolean {
|
||||||
return !!db().prepare('SELECT 1 FROM native_provisioning WHERE deployment_id = ?').get(id);
|
return !!db().prepare(`SELECT 1 FROM native_provisioning WHERE deployment_id = ?
|
||||||
|
UNION ALL SELECT 1 FROM native_creation_intents WHERE deployment_id = ?
|
||||||
|
UNION ALL SELECT 1 FROM native_creation_cancellations c
|
||||||
|
JOIN native_creation_intents i ON i.deployment_id=c.creation_id WHERE c.cleanup_id = ? LIMIT 1`).get(id, id, id);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function unplacedNativeCreation(server: string): string | null {
|
||||||
|
const row = db().prepare(`SELECT d.id FROM server_deployments d
|
||||||
|
JOIN native_creation_intents i ON i.deployment_id=d.id
|
||||||
|
LEFT JOIN native_provisioning n ON n.deployment_id=d.id
|
||||||
|
WHERE d.server=? AND d.server_deleted_at IS NULL AND d.action='create' AND d.state='queued'
|
||||||
|
AND n.deployment_id IS NULL ORDER BY d.rowid DESC LIMIT 1`).get(server) as { id: string } | undefined;
|
||||||
|
return row?.id ?? null;
|
||||||
}
|
}
|
||||||
|
|
||||||
const observations = new Map<string, Promise<unknown>>();
|
const observations = new Map<string, Promise<unknown>>();
|
||||||
|
|||||||
@@ -1,7 +1,12 @@
|
|||||||
import type Database from 'better-sqlite3';
|
import type Database from 'better-sqlite3';
|
||||||
|
|
||||||
export function migrateNativeInstances(handle: Database.Database): void {
|
export function migrateNativeInstances(handle: Database.Database): void {
|
||||||
|
migrateUnplacedCreationPort(handle);
|
||||||
handle.exec(`
|
handle.exec(`
|
||||||
|
CREATE TABLE IF NOT EXISTS native_creation_intents (
|
||||||
|
deployment_id TEXT PRIMARY KEY REFERENCES server_deployments(id) ON DELETE CASCADE,
|
||||||
|
selection TEXT NOT NULL
|
||||||
|
);
|
||||||
CREATE TABLE IF NOT EXISTS native_instances (
|
CREATE TABLE IF NOT EXISTS native_instances (
|
||||||
id TEXT PRIMARY KEY, server TEXT NOT NULL, host_id TEXT NOT NULL,
|
id TEXT PRIMARY KEY, server TEXT NOT NULL, host_id TEXT NOT NULL,
|
||||||
generation INTEGER NOT NULL DEFAULT 1, library_id TEXT NOT NULL,
|
generation INTEGER NOT NULL DEFAULT 1, library_id TEXT NOT NULL,
|
||||||
@@ -24,3 +29,27 @@ export function migrateNativeInstances(handle: Database.Database): void {
|
|||||||
);
|
);
|
||||||
`);
|
`);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Accepted native creations receive their host port only when provisioning places them. */
|
||||||
|
function migrateUnplacedCreationPort(handle: Database.Database): void {
|
||||||
|
const columns = handle.prepare('PRAGMA table_info(server_creation_jobs)').all() as { name: string; notnull: number }[];
|
||||||
|
if (!columns.find(column => column.name === 'port')?.notnull) return;
|
||||||
|
const { sql } = handle.prepare("SELECT sql FROM sqlite_master WHERE type='table' AND name='server_creation_jobs'").get() as { sql: string };
|
||||||
|
const replacement = sql.replace(/\bport\s+INTEGER\s+NOT\s+NULL\b/i, 'port INTEGER');
|
||||||
|
if (replacement === sql) throw new Error('Could not migrate unplaced creation ports.');
|
||||||
|
const dependent = handle.prepare("SELECT sql FROM sqlite_master WHERE tbl_name='server_creation_jobs' AND type IN ('index','trigger') AND sql IS NOT NULL")
|
||||||
|
.all() as { sql: string }[];
|
||||||
|
const foreignKeys = handle.pragma('foreign_keys', { simple: true });
|
||||||
|
handle.pragma('foreign_keys = OFF');
|
||||||
|
try {
|
||||||
|
handle.transaction(() => {
|
||||||
|
handle.exec(replacement.replace('server_creation_jobs', 'server_creation_jobs_unplaced'));
|
||||||
|
const names = columns.map(column => `"${column.name.replaceAll('"', '""')}"`).join(', ');
|
||||||
|
handle.exec(`INSERT INTO server_creation_jobs_unplaced(rowid,${names}) SELECT rowid,${names} FROM server_creation_jobs;
|
||||||
|
DROP TABLE server_creation_jobs;
|
||||||
|
ALTER TABLE server_creation_jobs_unplaced RENAME TO server_creation_jobs;`);
|
||||||
|
for (const object of dependent) handle.exec(object.sql);
|
||||||
|
if ((handle.pragma('foreign_key_check') as unknown[]).length) throw new Error('Unplaced creation migration found broken references.');
|
||||||
|
})();
|
||||||
|
} finally { handle.pragma(`foreign_keys = ${foreignKeys ? 'ON' : 'OFF'}`); }
|
||||||
|
}
|
||||||
|
|||||||
@@ -475,7 +475,7 @@ CREATE INDEX IF NOT EXISTS idx_server_deployment_events ON server_deployment_eve
|
|||||||
CREATE TABLE IF NOT EXISTS server_creation_jobs (
|
CREATE TABLE IF NOT EXISTS server_creation_jobs (
|
||||||
deployment_id TEXT PRIMARY KEY REFERENCES server_deployments(id) ON DELETE CASCADE,
|
deployment_id TEXT PRIMARY KEY REFERENCES server_deployments(id) ON DELETE CASCADE,
|
||||||
server TEXT NOT NULL,
|
server TEXT NOT NULL,
|
||||||
port INTEGER NOT NULL,
|
port INTEGER,
|
||||||
game_mode TEXT NOT NULL,
|
game_mode TEXT NOT NULL,
|
||||||
map_slug TEXT NOT NULL,
|
map_slug TEXT NOT NULL,
|
||||||
expected_map TEXT NOT NULL
|
expected_map TEXT NOT NULL
|
||||||
|
|||||||
@@ -4,18 +4,18 @@ import type { CollectionDetail, CollectionMode, CollectionPreview, CollectionSum
|
|||||||
import { db } from '../db/index.js';
|
import { db } from '../db/index.js';
|
||||||
import { config, komodoConfigured, rconConfigured } from '../config.js';
|
import { config, komodoConfigured, rconConfigured } from '../config.js';
|
||||||
import { blueprintEntries, summarizeRecipe, type BlueprintEntry } from './launch-recipes.js';
|
import { blueprintEntries, summarizeRecipe, type BlueprintEntry } from './launch-recipes.js';
|
||||||
import { creationQueue, destroyServer, nextAvailablePort, nextLobbyNumber, queueCreateServer } from '../provisioning/index.js';
|
import { creationQueue, destroyServer, nextAvailablePort, queueCreateServer } from '../provisioning/index.js';
|
||||||
import { deploymentStage, finishDeployment, recordDeploymentEvent } from './deployments.js';
|
import { deploymentStage, finishDeployment, recordDeploymentEvent } from './deployments.js';
|
||||||
import { housePartyTargets } from './house-party.js';
|
import { housePartyTargets } from './house-party.js';
|
||||||
import { getServerKey } from './server-keys.js';
|
|
||||||
import { serverObservation } from './status.js';
|
import { serverObservation } from './status.js';
|
||||||
import { activeBetaRemovalBlocker, getServer } from './servers.js';
|
import { activeBetaRemovalBlocker, getServer } from './servers.js';
|
||||||
import { parseServerName, zombieMapBySlug, zombieMaps } from './map-aliases.js';
|
import { parseServerName, zombieMapBySlug, zombieMaps } from './map-aliases.js';
|
||||||
import { nativeSelection, nextNativePort } from '../provisioning/native.js';
|
import { nativeLaunchSelection, queueNativeCreationIntent } from '../provisioning/native.js';
|
||||||
import { listHosts } from '../hosts/index.js';
|
import { listHosts } from '../hosts/index.js';
|
||||||
import { nativeInstance } from '../backends/native.js';
|
import { nativeDeployment, nativeInstance } from '../backends/native.js';
|
||||||
|
import { launchAvailability } from '../hosts/availability.js';
|
||||||
|
|
||||||
function coverage(entries: BlueprintEntry[]) {
|
function coverage(entries: BlueprintEntry[], includeHealth = true) {
|
||||||
type Server = { name: string; game: string | null; map_slug: string | null; map: string | null };
|
type Server = { name: string; game: string | null; map_slug: string | null; map: string | null };
|
||||||
const registered = db().prepare('SELECT name, game, map_slug, map FROM servers WHERE destroyed_at IS NULL').all() as Server[];
|
const registered = db().prepare('SELECT name, game, map_slug, map FROM servers WHERE destroyed_at IS NULL').all() as Server[];
|
||||||
const queued = db().prepare(`SELECT j.server AS name, substr(j.server, 1, 2) AS game, j.map_slug, j.expected_map AS map
|
const queued = db().prepare(`SELECT j.server AS name, substr(j.server, 1, 2) AS game, j.map_slug, j.expected_map AS map
|
||||||
@@ -37,14 +37,14 @@ function coverage(entries: BlueprintEntry[]) {
|
|||||||
const matches = zombieMaps(game).filter(map => map.canonical === server.map);
|
const matches = zombieMaps(game).filter(map => map.canonical === server.map);
|
||||||
return matches.length === 1 && matches[0].slug === entry.recipe.mapSlug;
|
return matches.length === 1 && matches[0].slug === entry.recipe.mapSlug;
|
||||||
}).map(server => server.name).sort();
|
}).map(server => server.name).sort();
|
||||||
const coverage = existingServers.map(name => {
|
const coverage = includeHealth ? existingServers.map(name => {
|
||||||
const operation = db().prepare("SELECT id, state FROM server_deployments WHERE server = ? AND server_deleted_at IS NULL AND action IN ('create','rebuild') AND state IN ('queued', 'running', 'awaiting-health') ORDER BY rowid DESC LIMIT 1").get(name) as { id: string; state: string } | undefined;
|
const operation = db().prepare("SELECT id, state FROM server_deployments WHERE server = ? AND server_deleted_at IS NULL AND action IN ('create','rebuild') AND state IN ('queued', 'running', 'awaiting-health') ORDER BY rowid DESC LIMIT 1").get(name) as { id: string; state: string } | undefined;
|
||||||
const server = getServer(name);
|
const server = getServer(name);
|
||||||
return { name, health: operation ? operation.state === 'queued' ? 'queued' : 'starting'
|
return { name, health: operation ? operation.state === 'queued' ? 'queued' : 'starting'
|
||||||
: server?.container_state && ['stopped', 'exited', 'created', 'dead'].includes(server.container_state) ? 'stopped'
|
: server?.container_state && ['stopped', 'exited', 'created', 'dead'].includes(server.container_state) ? 'stopped'
|
||||||
: server ? serverObservation(server).health : 'offline',
|
: server ? serverObservation(server).health : 'offline',
|
||||||
href: operation ? `/provisioning/${encodeURIComponent(operation.id)}` : `/servers/${encodeURIComponent(name)}` };
|
href: operation ? `/provisioning/${encodeURIComponent(operation.id)}` : `/servers/${encodeURIComponent(name)}` };
|
||||||
});
|
}) : [];
|
||||||
return { existingServers, coverage, existing: existingServers.length, missing: Math.max(0, entry.count - existingServers.length) };
|
return { existingServers, coverage, existing: existingServers.length, missing: Math.max(0, entry.count - existingServers.length) };
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
@@ -57,8 +57,8 @@ function fleetSnapshot() {
|
|||||||
}));
|
}));
|
||||||
}
|
}
|
||||||
|
|
||||||
function collectionPlan(entries: BlueprintEntry[], mode: CollectionMode) {
|
function collectionPlan(entries: BlueprintEntry[], mode: CollectionMode, includeHealth = true) {
|
||||||
const counts = coverage(entries);
|
const counts = coverage(entries, includeHealth);
|
||||||
// Preserve deployed lobbies before queued ones, then the lowest lobby IDs.
|
// Preserve deployed lobbies before queued ones, then the lowest lobby IDs.
|
||||||
const retainedServers = counts.flatMap((entry, index) => [...entry.existingServers].sort((a, b) =>
|
const retainedServers = counts.flatMap((entry, index) => [...entry.existingServers].sort((a, b) =>
|
||||||
Number(!!getServer(b)) - Number(!!getServer(a)) || a.localeCompare(b, 'en', { numeric: true })
|
Number(!!getServer(b)) - Number(!!getServer(a)) || a.localeCompare(b, 'en', { numeric: true })
|
||||||
@@ -71,50 +71,56 @@ function collectionPlan(entries: BlueprintEntry[], mode: CollectionMode) {
|
|||||||
return { counts, retainedServers, removals };
|
return { counts, retainedServers, removals };
|
||||||
}
|
}
|
||||||
|
|
||||||
function allocate(entries: BlueprintEntry[], releasingServers: string[] = []): CollectionPreview['servers'] {
|
function allocate(entries: BlueprintEntry[], releasingServers: string[] = [], counts = coverage(entries)): CollectionPreview['servers'] {
|
||||||
const names: string[] = [];
|
const occupied = db().prepare(`SELECT name FROM servers UNION
|
||||||
|
SELECT j.server AS name FROM server_creation_jobs j JOIN server_deployments d ON d.id=j.deployment_id
|
||||||
|
WHERE d.server_deleted_at IS NULL AND (d.state IN ('queued','running','awaiting-health')
|
||||||
|
OR (d.action='rebuild' AND (d.state='waiting' OR (d.state IN ('failed','interrupted') AND d.stage='Removing server files and registry'))))`)
|
||||||
|
.all() as { name: string }[];
|
||||||
|
const lobbies = new Map<string, Set<number>>();
|
||||||
|
for (const { name } of occupied) {
|
||||||
|
const match = /^(.*)_(\d+)$/.exec(name);
|
||||||
|
if (!match) continue;
|
||||||
|
const taken = lobbies.get(match[1]) ?? new Set<number>();
|
||||||
|
taken.add(Number(match[2])); lobbies.set(match[1], taken);
|
||||||
|
}
|
||||||
|
const savedKeys = new Set((db().prepare("SELECT server_id FROM server_keys WHERE length(trim(pluto_key))>0").all() as { server_id: string }[]).map(row => row.server_id));
|
||||||
const ports: number[] = [];
|
const ports: number[] = [];
|
||||||
const planned: { hostId: string; port: number }[] = [];
|
|
||||||
const counts = coverage(entries);
|
|
||||||
return entries.flatMap((entry, entryIndex) => Array.from({ length: counts[entryIndex].missing }, () => {
|
return entries.flatMap((entry, entryIndex) => Array.from({ length: counts[entryIndex].missing }, () => {
|
||||||
const base = `${entry.recipe.game}_zombie_${entry.recipe.mapSlug}`;
|
const base = `${entry.recipe.game}_zombie_${entry.recipe.mapSlug}`;
|
||||||
const lobby = nextLobbyNumber(base, names);
|
const taken = lobbies.get(base) ?? new Set<number>();
|
||||||
|
const lobby = Array.from({ length: 20 }, (_, index) => index + 1).find(number => !taken.has(number)) ?? null;
|
||||||
const name = lobby === null ? null : `${base}_${lobby}`;
|
const name = lobby === null ? null : `${base}_${lobby}`;
|
||||||
let native = entry.recipe.native;
|
const native = entry.recipe.native;
|
||||||
let port: number | null;
|
const port = native ? null : nextAvailablePort(ports, releasingServers);
|
||||||
if (native) {
|
if (lobby !== null) { taken.add(lobby); lobbies.set(base, taken); }
|
||||||
try {
|
|
||||||
native = nativeSelection(native as unknown as Record<string, unknown>, entry.recipe.game, { planned });
|
|
||||||
port = nextNativePort(native, planned.filter(item => item.hostId === native!.hostId).map(item => item.port));
|
|
||||||
if (port !== null) planned.push({ hostId: native.hostId, port });
|
|
||||||
} catch { port = null; }
|
|
||||||
} else port = nextAvailablePort(ports, releasingServers);
|
|
||||||
if (name) names.push(name);
|
|
||||||
if (port !== null) ports.push(port);
|
if (port !== null) ports.push(port);
|
||||||
return { entryIndex, name, port, ...(native ? { native } : {}), hasSavedKey: !!name && !!getServerKey(name) };
|
return { entryIndex, name, port, ...(native ? { native } : {}), hasSavedKey: !!name && savedKeys.has(name) };
|
||||||
}));
|
}));
|
||||||
}
|
}
|
||||||
|
|
||||||
function blockersFor(servers: CollectionPreview['servers'], entries: BlueprintEntry[] = []): string[] {
|
function blockersFor(servers: CollectionPreview['servers'], entries: BlueprintEntry[] = []): string[] {
|
||||||
const blockers: string[] = [];
|
const blockers: string[] = [];
|
||||||
if (!servers.length) return blockers;
|
if (!servers.length) return blockers;
|
||||||
if (entries.some(entry => entry.recipe.native)) {
|
const legacy = servers.filter(server => !server.native && !entries[server.entryIndex]?.recipe.native);
|
||||||
for (const entry of entries) {
|
if (legacy.length) {
|
||||||
try { nativeSelection(entry.recipe.native as unknown as Record<string, unknown>, entry.recipe.game); }
|
|
||||||
catch (error) { blockers.push((error as Error).message); }
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if (!komodoConfigured() || !config.komodo.targetServer) blockers.push('Select and prepare a host before launching.');
|
if (!komodoConfigured() || !config.komodo.targetServer) blockers.push('Select and prepare a host before launching.');
|
||||||
if (!rconConfigured()) blockers.push('Configure RCON_PASSWORD so every new server can be verified.');
|
if (!rconConfigured()) blockers.push('Configure RCON_PASSWORD so every new server can be verified.');
|
||||||
}
|
}
|
||||||
if (servers.some(server => !server.name)) blockers.push('There are not enough free lobbies for one or more maps (20 per map). Reduce their counts or free capacity.');
|
if (servers.some(server => !server.name)) blockers.push('There are not enough free lobbies for one or more maps (20 per map). Reduce their counts or free capacity.');
|
||||||
if (servers.some(server => server.port === null)) blockers.push(entries.some(entry => entry.recipe.native) ? 'There is not enough host CPU, memory or port capacity for the whole collection.' : `There are not enough free game ports in ${config.portRange.min}–${config.portRange.max} for the whole collection.`);
|
if (legacy.some(server => server.port === null)) blockers.push(`There are not enough free game ports in ${config.portRange.min}–${config.portRange.max} for the whole collection.`);
|
||||||
return blockers;
|
return blockers;
|
||||||
}
|
}
|
||||||
|
|
||||||
function removalBlockersFor(removals: CollectionPreview['removals']): string[] {
|
function removalBlockersFor(removals: CollectionPreview['removals']): string[] {
|
||||||
const blockers: string[] = [];
|
const blockers: string[] = [];
|
||||||
if (removals.some(server => !nativeInstance(server.name)) && (!komodoConfigured() || !rconConfigured())) blockers.push('Configure the legacy backend before removing its servers.');
|
const native = (name: string) => {
|
||||||
|
if (nativeInstance(name)) return true;
|
||||||
|
const creation = db().prepare(`SELECT id FROM server_deployments WHERE server=? AND server_deleted_at IS NULL
|
||||||
|
AND action='create' AND state IN ('queued','running','awaiting-health') ORDER BY rowid DESC LIMIT 1`).get(name) as { id: string } | undefined;
|
||||||
|
return !!creation && nativeDeployment(creation.id);
|
||||||
|
};
|
||||||
|
if (removals.some(server => !native(server.name)) && (!komodoConfigured() || !rconConfigured())) blockers.push('Configure the legacy backend before removing its servers.');
|
||||||
for (const server of removals) {
|
for (const server of removals) {
|
||||||
const blocker = activeBetaRemovalBlocker(server.name);
|
const blocker = activeBetaRemovalBlocker(server.name);
|
||||||
if (blocker) blockers.push(blocker);
|
if (blocker) blockers.push(blocker);
|
||||||
@@ -130,24 +136,26 @@ export function previewCollection(blueprintId: unknown, mode: unknown = 'fill',
|
|||||||
if (selection?.hostId !== undefined || !config.komodo.address || listHosts().length > 0) {
|
if (selection?.hostId !== undefined || !config.komodo.address || listHosts().length > 0) {
|
||||||
const libraries = selection?.libraryIds as Record<string, unknown> | undefined;
|
const libraries = selection?.libraryIds as Record<string, unknown> | undefined;
|
||||||
for (const entry of entries) {
|
for (const entry of entries) {
|
||||||
entry.recipe.native = nativeSelection({ ...selection, libraryId: libraries?.[entry.recipe.game] ?? selection?.libraryId }, entry.recipe.game);
|
entry.recipe.native = nativeLaunchSelection({ ...selection, libraryId: libraries?.[entry.recipe.game] ?? selection?.libraryId }, entry.recipe.game);
|
||||||
entry.recipe.image = entry.recipe.native.image!;
|
entry.recipe.image = entry.recipe.native.image!;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
const { counts, retainedServers, removals } = collectionPlan(entries, mode);
|
const { counts, retainedServers, removals } = collectionPlan(entries, mode);
|
||||||
const servers = allocate(entries, removals.map(server => server.name));
|
const servers = allocate(entries, removals.map(server => server.name), counts);
|
||||||
const blockers = [...blockersFor(servers, entries), ...removalBlockersFor(removals)];
|
const blockers = [...blockersFor(servers, entries), ...removalBlockersFor(removals)];
|
||||||
|
const selections = servers.flatMap(server => server.native ? [server.native] : []);
|
||||||
|
const availability = selections.length ? launchAvailability(selections) : {};
|
||||||
const id = randomUUID();
|
const id = randomUUID();
|
||||||
const expiresAt = new Date(Date.now() + 30 * 60_000).toISOString();
|
const expiresAt = new Date(Date.now() + 30 * 60_000).toISOString();
|
||||||
db().prepare('DELETE FROM server_collection_previews WHERE expires_at < ? AND collection_id IS NULL').run(new Date().toISOString());
|
db().prepare('DELETE FROM server_collection_previews WHERE expires_at < ? AND collection_id IS NULL').run(new Date().toISOString());
|
||||||
db().prepare('INSERT INTO server_collection_previews (id, expires_at, blueprint_name, entries, servers, mode, fleet_snapshot) VALUES (?, ?, ?, ?, ?, ?, ?)')
|
db().prepare('INSERT INTO server_collection_previews (id, expires_at, blueprint_name, entries, servers, mode, fleet_snapshot) VALUES (?, ?, ?, ?, ?, ?, ?)')
|
||||||
.run(id, expiresAt, blueprintName, JSON.stringify(entries), JSON.stringify(servers), mode, mode === 'exact' ? JSON.stringify(fleetSnapshot()) : null);
|
.run(id, expiresAt, blueprintName, JSON.stringify(entries), JSON.stringify(servers), mode, mode === 'exact' ? JSON.stringify(fleetSnapshot()) : null);
|
||||||
return { id, expiresAt, blueprintName, mode, retainedServers, removals, launchConcurrency: fleetSettings().createConcurrency, servers, blockers, ready: blockers.length === 0,
|
return { id, expiresAt, blueprintName, mode, retainedServers, removals, launchConcurrency: fleetSettings().createConcurrency, servers, blockers, ...availability, ready: blockers.length === 0,
|
||||||
entries: entries.map((entry, index) => ({ count: entry.count, ...counts[index], recipe: summarizeRecipe(entry.recipe) })) };
|
entries: entries.map((entry, index) => ({ count: entry.count, ...counts[index], recipe: summarizeRecipe(entry.recipe) })) };
|
||||||
})();
|
})();
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Reserve and enqueue the entire collection atomically. No external operations here. */
|
/** Save all reviewed identities and keys atomically; native placement belongs to provisioning. */
|
||||||
export function queueCollection(input: { previewId?: unknown; keys?: unknown }): { ok: true; collectionId: string | null } {
|
export function queueCollection(input: { previewId?: unknown; keys?: unknown }): { ok: true; collectionId: string | null } {
|
||||||
if (typeof input.previewId !== 'string' || !input.previewId) throw new Error('Review a collection preview before launching.');
|
if (typeof input.previewId !== 'string' || !input.previewId) throw new Error('Review a collection preview before launching.');
|
||||||
if (input.keys !== undefined && (!input.keys || typeof input.keys !== 'object' || Array.isArray(input.keys))) throw new Error('Provide keys indexed by the exact server ID.');
|
if (input.keys !== undefined && (!input.keys || typeof input.keys !== 'object' || Array.isArray(input.keys))) throw new Error('Provide keys indexed by the exact server ID.');
|
||||||
@@ -160,19 +168,35 @@ export function queueCollection(input: { previewId?: unknown; keys?: unknown }):
|
|||||||
if (preview.collection_id) return { ok: true as const, collectionId: preview.collection_id };
|
if (preview.collection_id) return { ok: true as const, collectionId: preview.collection_id };
|
||||||
if (preview.expires_at <= new Date().toISOString()) throw new Error('Collection preview expired. Refresh it before launching.');
|
if (preview.expires_at <= new Date().toISOString()) throw new Error('Collection preview expired. Refresh it before launching.');
|
||||||
const entries: BlueprintEntry[] = JSON.parse(preview.entries);
|
const entries: BlueprintEntry[] = JSON.parse(preview.entries);
|
||||||
// Older previews embedded plugin bytes. Child previews need only the
|
// Older previews embedded plugin bytes. Queued jobs need only the
|
||||||
// reviewed metadata; provisioning captures the current files at dispatch.
|
// reviewed metadata; provisioning captures the current files at dispatch.
|
||||||
for (const entry of entries) entry.recipe.files = entry.recipe.files.map(({ content: _content, ...file }) => file);
|
for (const entry of entries) entry.recipe.files = entry.recipe.files.map(({ content: _content, ...file }) => file);
|
||||||
const reviewed: CollectionPreview['servers'] = JSON.parse(preview.servers);
|
const reviewed: CollectionPreview['servers'] = JSON.parse(preview.servers);
|
||||||
if (preview.mode === 'exact' && preview.fleet_snapshot !== JSON.stringify(fleetSnapshot())) {
|
if (preview.mode === 'exact' && preview.fleet_snapshot !== JSON.stringify(fleetSnapshot())) {
|
||||||
throw new Error('The fleet changed. Nothing was queued. Refresh the preview and review the servers to keep and remove.');
|
throw new Error('The fleet changed. Nothing was queued. Refresh the preview and review the servers to keep and remove.');
|
||||||
}
|
}
|
||||||
const { retainedServers, removals } = collectionPlan(entries, preview.mode);
|
const { counts, retainedServers, removals } = collectionPlan(entries, preview.mode, false);
|
||||||
const releasingServers = removals.map(server => server.name);
|
const releasingServers = removals.map(server => server.name);
|
||||||
const servers = allocate(entries, releasingServers);
|
const nativeReview = reviewed.every(server => server.native || entries[server.entryIndex]?.recipe.native);
|
||||||
|
// Native Apply accepts the reviewed logical identities. Capacity and
|
||||||
|
// physical ports can change while this request waits in the queue.
|
||||||
|
const servers = nativeReview ? reviewed : allocate(entries, releasingServers, counts);
|
||||||
|
const validatedSelections = new Set<string>();
|
||||||
|
for (const server of servers) {
|
||||||
|
const recipe = entries[server.entryIndex].recipe;
|
||||||
|
const selection = server.native ?? recipe.native;
|
||||||
|
if (!selection) continue;
|
||||||
|
const identity = JSON.stringify([recipe.game, selection]);
|
||||||
|
if (validatedSelections.has(identity)) continue;
|
||||||
|
// Recheck saved configuration once per selection without allocating
|
||||||
|
// physical capacity or changing the names and files already reviewed.
|
||||||
|
nativeLaunchSelection({ ...selection }, recipe.game);
|
||||||
|
validatedSelections.add(identity);
|
||||||
|
}
|
||||||
const blockers = [...blockersFor(servers, entries), ...removalBlockersFor(removals)];
|
const blockers = [...blockersFor(servers, entries), ...removalBlockersFor(removals)];
|
||||||
if (blockers.length) throw new Error(blockers.join(' '));
|
if (blockers.length) throw new Error(blockers.join(' '));
|
||||||
if (servers.length !== reviewed.length || servers.some((server, i) => server.entryIndex !== reviewed[i]?.entryIndex || server.name !== reviewed[i]?.name || (!server.native?.automatic && server.port !== reviewed[i]?.port))) {
|
if (entries.some((_entry, index) => counts[index].missing !== reviewed.filter(server => server.entryIndex === index).length)
|
||||||
|
|| servers.length !== reviewed.length || servers.some((server, i) => server.entryIndex !== reviewed[i]?.entryIndex || server.name !== reviewed[i]?.name || (!server.native && server.port !== reviewed[i]?.port))) {
|
||||||
throw new Error('Fleet counts, lobby or port availability changed. Nothing was queued. Refresh the preview and review each server key.');
|
throw new Error('Fleet counts, lobby or port availability changed. Nothing was queued. Refresh the preview and review each server key.');
|
||||||
}
|
}
|
||||||
const expected = new Set(servers.map(server => server.name));
|
const expected = new Set(servers.map(server => server.name));
|
||||||
@@ -210,16 +234,21 @@ export function queueCollection(input: { previewId?: unknown; keys?: unknown }):
|
|||||||
});
|
});
|
||||||
servers.forEach((server, index) => {
|
servers.forEach((server, index) => {
|
||||||
const recipe = { ...entries[server.entryIndex].recipe, ...(server.native ? { native: server.native } : {}) };
|
const recipe = { ...entries[server.entryIndex].recipe, ...(server.native ? { native: server.native } : {}) };
|
||||||
|
let result;
|
||||||
|
if (recipe.native) result = queueNativeCreationIntent({ server: server.name!, game: recipe.game, mapSlug: recipe.mapSlug,
|
||||||
|
selection: recipe.native, recipe, plutoKey: keys[server.name!] as string | undefined, createdBy: 'ui', blueprintName: preview.blueprint_name });
|
||||||
|
else {
|
||||||
const childPreviewId = randomUUID();
|
const childPreviewId = randomUUID();
|
||||||
db().prepare('INSERT INTO server_launch_previews (id, expires_at, server, port, recipe, blueprint_name) VALUES (?, ?, ?, ?, ?, ?)')
|
db().prepare('INSERT INTO server_launch_previews (id, expires_at, server, port, recipe, blueprint_name) VALUES (?, ?, ?, ?, ?, ?)')
|
||||||
.run(childPreviewId, preview.expires_at, server.name, server.port, JSON.stringify(recipe), preview.blueprint_name);
|
.run(childPreviewId, preview.expires_at, server.name, server.port, JSON.stringify(recipe), preview.blueprint_name);
|
||||||
const result = queueCreateServer({ game: recipe.game, gameMode: 'zombie', mapSlug: recipe.mapSlug,
|
result = queueCreateServer({ game: recipe.game, gameMode: 'zombie', mapSlug: recipe.mapSlug,
|
||||||
expectedName: server.name!, previewId: childPreviewId, plutoKey: keys[server.name!] as string | undefined, createdBy: 'ui' }, releasingServers);
|
expectedName: server.name!, previewId: childPreviewId, plutoKey: keys[server.name!] as string | undefined, createdBy: 'ui' }, releasingServers);
|
||||||
|
}
|
||||||
// Throw so the outer transaction also reverses all earlier keys and children.
|
// Throw so the outer transaction also reverses all earlier keys and children.
|
||||||
if (!result.ok) throw new Error(`${server.name}: ${result.message} Nothing was queued.`);
|
if (!result.ok) throw new Error(`${server.name}: ${result.message} Nothing was queued.`);
|
||||||
db().prepare('INSERT INTO server_collection_members (collection_id, deployment_id, position) VALUES (?, ?, ?)')
|
db().prepare('INSERT INTO server_collection_members (collection_id, deployment_id, position) VALUES (?, ?, ?)')
|
||||||
.run(collectionId, result.deploymentId, index);
|
.run(collectionId, result.deploymentId, index);
|
||||||
for (const removal of removalJobs.filter(removal => removal.ports.includes(server.port))) {
|
for (const removal of recipe.native ? [] : removalJobs.filter(removal => removal.ports.includes(server.port))) {
|
||||||
db().prepare('INSERT INTO server_creation_dependencies (deployment_id, removal_id) VALUES (?, ?)').run(result.deploymentId, removal.id);
|
db().prepare('INSERT INTO server_creation_dependencies (deployment_id, removal_id) VALUES (?, ?)').run(result.deploymentId, removal.id);
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -439,6 +439,18 @@ function libraryInUse(versionId: string, hostId?: string): boolean {
|
|||||||
if (!handle.prepare("SELECT 1 FROM sqlite_master WHERE type='table' AND name='native_instances'").get()) return true;
|
if (!handle.prepare("SELECT 1 FROM sqlite_master WHERE type='table' AND name='native_instances'").get()) return true;
|
||||||
if (handle.prepare(`SELECT 1 FROM native_instances WHERE library_id=? AND retired_at IS NULL ${hostId ? 'AND host_id=?' : ''} LIMIT 1`)
|
if (handle.prepare(`SELECT 1 FROM native_instances WHERE library_id=? AND retired_at IS NULL ${hostId ? 'AND host_id=?' : ''} LIMIT 1`)
|
||||||
.get(versionId, ...(hostId ? [hostId] : []))) return true;
|
.get(versionId, ...(hostId ? [hostId] : []))) return true;
|
||||||
|
// Accepted jobs pin their game files before placement creates an instance.
|
||||||
|
// Automatic intents may still choose any host, so retain each available copy.
|
||||||
|
if (handle.prepare("SELECT 1 FROM sqlite_master WHERE type='table' AND name='native_creation_intents'").get()
|
||||||
|
&& handle.prepare(`SELECT 1 FROM server_deployments d
|
||||||
|
JOIN native_creation_intents q ON q.deployment_id=d.id
|
||||||
|
LEFT JOIN native_provisioning n ON n.deployment_id=d.id
|
||||||
|
WHERE d.action='create' AND d.server_deleted_at IS NULL
|
||||||
|
AND d.state IN ('queued','waiting','running','awaiting-health','interrupted') AND n.deployment_id IS NULL
|
||||||
|
AND json_extract(q.selection,'$.libraryId')=?
|
||||||
|
${hostId ? `AND (json_extract(q.selection,'$.automatic')=1 OR json_extract(q.selection,'$.hostId') IS NULL
|
||||||
|
OR json_extract(q.selection,'$.hostId')=?)` : ''} LIMIT 1`)
|
||||||
|
.get(versionId, ...(hostId ? [hostId] : []))) return true;
|
||||||
return !!handle.prepare(`SELECT 1 FROM ec_host_operations WHERE state IN ('pending','accepted','running','unknown')
|
return !!handle.prepare(`SELECT 1 FROM ec_host_operations WHERE state IN ('pending','accepted','running','unknown')
|
||||||
AND (json_extract(payload,'$.libraryVersionId')=? OR json_extract(payload,'$.libraryId')=? OR json_extract(payload,'$.importId')=?)
|
AND (json_extract(payload,'$.libraryVersionId')=? OR json_extract(payload,'$.libraryId')=? OR json_extract(payload,'$.importId')=?)
|
||||||
${hostId ? 'AND host_id=?' : ''} LIMIT 1`).get(versionId, versionId, versionId, ...(hostId ? [hostId] : []));
|
${hostId ? 'AND host_id=?' : ''} LIMIT 1`).get(versionId, versionId, versionId, ...(hostId ? [hostId] : []));
|
||||||
|
|||||||
@@ -3,6 +3,11 @@ import type Database from 'better-sqlite3';
|
|||||||
const initialized = new WeakSet<Database.Database>();
|
const initialized = new WeakSet<Database.Database>();
|
||||||
export function ensureGameFileSchema(handle: Database.Database): void {
|
export function ensureGameFileSchema(handle: Database.Database): void {
|
||||||
if (initialized.has(handle)) return;
|
if (initialized.has(handle)) return;
|
||||||
|
// Worker readers open only after the controller has initialized the schema.
|
||||||
|
if (handle.readonly) {
|
||||||
|
initialized.add(handle);
|
||||||
|
return;
|
||||||
|
}
|
||||||
handle.exec(`
|
handle.exec(`
|
||||||
CREATE TABLE IF NOT EXISTS game_file_libraries (
|
CREATE TABLE IF NOT EXISTS game_file_libraries (
|
||||||
id TEXT PRIMARY KEY, name TEXT NOT NULL COLLATE NOCASE UNIQUE,
|
id TEXT PRIMARY KEY, name TEXT NOT NULL COLLATE NOCASE UNIQUE,
|
||||||
|
|||||||
@@ -0,0 +1,141 @@
|
|||||||
|
import { db } from '../db/index.js';
|
||||||
|
import { fleetSettings } from '../settings.js';
|
||||||
|
import { subscribe } from '../pipeline/bus.js';
|
||||||
|
import { capacityCalibration } from './calibration.js';
|
||||||
|
import { HEARTBEAT_MAX_AGE_MS, listHosts } from './index.js';
|
||||||
|
import type { Host, NativePlacement } from '../../hosts.js';
|
||||||
|
import type { LaunchAvailability } from '../../deployment-plan.js';
|
||||||
|
|
||||||
|
interface HostAvailability {
|
||||||
|
host: Host;
|
||||||
|
freeCpu: number | null;
|
||||||
|
freeMemory: number | null;
|
||||||
|
runningCpu: number;
|
||||||
|
runningMemory: number;
|
||||||
|
ports: { port: number; bind_address: string }[];
|
||||||
|
}
|
||||||
|
interface Snapshot {
|
||||||
|
updatedAt: string;
|
||||||
|
hosts: Map<string, HostAvailability>;
|
||||||
|
startupLimit: number;
|
||||||
|
active: number;
|
||||||
|
queued: number;
|
||||||
|
}
|
||||||
|
const snapshots = new WeakMap<ReturnType<typeof db>, Snapshot>();
|
||||||
|
const scheduled = new WeakSet<ReturnType<typeof db>>();
|
||||||
|
|
||||||
|
function snapshotSchemaReady(handle: ReturnType<typeof db>): boolean {
|
||||||
|
// Host enrollment and game-file maintenance can run before provisioning is
|
||||||
|
// initialized. Missing queue data means unknown capacity, not an empty fleet.
|
||||||
|
const { count } = handle.prepare(`SELECT COUNT(*) AS count FROM sqlite_master WHERE type='table'
|
||||||
|
AND name IN ('native_instances','native_provisioning','server_deployments','settings')`).get() as { count: number };
|
||||||
|
if (count === 4) return true;
|
||||||
|
snapshots.delete(handle);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
function queueSnapshot(handle: ReturnType<typeof db>, hosts: Map<string, HostAvailability>): Snapshot {
|
||||||
|
const counts = handle.prepare(`SELECT
|
||||||
|
COALESCE(SUM(state IN ('running','awaiting-health')),0) AS active,
|
||||||
|
COALESCE(SUM(state='queued'),0) AS queued FROM server_deployments
|
||||||
|
WHERE action IN ('create','rebuild') AND state IN ('queued','running','awaiting-health')
|
||||||
|
AND server_deleted_at IS NULL`).get() as { active: number; queued: number };
|
||||||
|
return { hosts, ...counts, startupLimit: fleetSettings().createConcurrency, updatedAt: new Date().toISOString() };
|
||||||
|
}
|
||||||
|
|
||||||
|
function observeHost(host: Host): HostAvailability {
|
||||||
|
const handle = db();
|
||||||
|
const { running, startup } = capacityCalibration(host.id);
|
||||||
|
const measured = new Map(host.inventory.instanceResources?.map(sample => [sample.instanceId, sample.generation]));
|
||||||
|
const pending = handle.prepare(`SELECT DISTINCT i.id,i.generation FROM native_instances i
|
||||||
|
JOIN native_provisioning n ON n.instance_id=i.id JOIN server_deployments d ON d.id=n.deployment_id
|
||||||
|
WHERE i.host_id=? AND i.retired_at IS NULL AND d.action IN ('create','rebuild')
|
||||||
|
AND d.state IN ('running','awaiting-health')`).all(host.id) as { id: string; generation: number }[];
|
||||||
|
const unmeasured = pending.filter(instance => measured.get(instance.id) !== instance.generation).length;
|
||||||
|
const { cpuCount, cpuUsagePercent, memoryTotal, memoryAvailable } = host.inventory;
|
||||||
|
const freeCpu = cpuCount && cpuUsagePercent !== undefined
|
||||||
|
? Math.max(0, cpuCount * (0.95 - cpuUsagePercent / 100) - unmeasured * startup.cpuCores) : null;
|
||||||
|
const freeMemory = memoryTotal && memoryAvailable !== undefined && memoryAvailable <= memoryTotal
|
||||||
|
? Math.max(0, memoryAvailable - 500_000_000 - unmeasured * startup.memoryBytes) : null;
|
||||||
|
return { host, freeCpu, freeMemory, runningCpu: running.cpuCores, runningMemory: running.memoryBytes,
|
||||||
|
ports: handle.prepare('SELECT port,bind_address FROM ec_host_ports WHERE host_id=?').all(host.id) as HostAvailability['ports'] };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Host telemetry updates the estimate once, independently of how many lobbies are previewed. */
|
||||||
|
export function refreshHostAvailability(host: Host): void {
|
||||||
|
const handle = db();
|
||||||
|
if (!snapshotSchemaReady(handle)) return;
|
||||||
|
const hosts = new Map(snapshots.get(handle)?.hosts);
|
||||||
|
hosts.set(host.id, observeHost(host));
|
||||||
|
snapshots.set(handle, queueSnapshot(handle, hosts));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Lifecycle events refresh the shared snapshot after their transaction and request have yielded. */
|
||||||
|
export function scheduleAvailabilityRefresh(): void {
|
||||||
|
const handle = db();
|
||||||
|
if (scheduled.has(handle)) return;
|
||||||
|
scheduled.add(handle);
|
||||||
|
setImmediate(() => {
|
||||||
|
scheduled.delete(handle);
|
||||||
|
if (!handle.open) return;
|
||||||
|
try {
|
||||||
|
if (!snapshotSchemaReady(handle)) return;
|
||||||
|
const hosts = new Map(listHosts().map(host => [host.id, observeHost(host)]));
|
||||||
|
snapshots.set(handle, queueSnapshot(handle, hosts));
|
||||||
|
} catch (error) { console.error('[availability] snapshot refresh failed:', error); }
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
subscribe(event => {
|
||||||
|
if (event.type === 'fleet-invalidated' || event.type === 'server-status') scheduleAvailabilityRefresh();
|
||||||
|
});
|
||||||
|
|
||||||
|
function freePorts(snapshot: HostAvailability, bindAddress?: string): number {
|
||||||
|
const bind = bindAddress ?? snapshot.host.bindAddress;
|
||||||
|
const used = new Set(snapshot.ports.filter(row => row.port >= snapshot.host.portMin && row.port <= snapshot.host.portMax
|
||||||
|
&& (row.bind_address === bind || ['0.0.0.0', '::'].includes(bind) || ['0.0.0.0', '::'].includes(row.bind_address))).map(row => row.port));
|
||||||
|
return Math.max(0, snapshot.host.portMax - snapshot.host.portMin + 1 - used.size);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Estimates only: no placement, historical scans, reservations, or live host requests. */
|
||||||
|
export function launchAvailability(selections: NativePlacement[]): { availability: LaunchAvailability; warnings: string[] } {
|
||||||
|
const snapshot = snapshots.get(db());
|
||||||
|
if (!snapshot) {
|
||||||
|
scheduleAvailabilityRefresh();
|
||||||
|
return { availability: { updatedAt: null, requested: selections.length, estimatedAvailable: null, portsAvailable: null,
|
||||||
|
cpuCoresAvailable: null, memoryBytesAvailable: null, startupSlots: null, startupLimit: null, queued: null },
|
||||||
|
warnings: selections.length ? ['Capacity measurements are updating. You can queue this launch; provisioning waits for an eligible host.'] : [] };
|
||||||
|
}
|
||||||
|
const candidates = [...snapshot.hosts.values()].filter(({ host }) => host.readiness === 'ready'
|
||||||
|
&& host.lastHeartbeat && Date.now() - Date.parse(host.lastHeartbeat) <= HEARTBEAT_MAX_AGE_MS
|
||||||
|
&& (host.inventory.storageAvailable ?? 0) >= host.minimumFreeBytes + 1024 ** 3);
|
||||||
|
const eligible = (candidate: HostAvailability, selection: NativePlacement) => (!selection.hostId || selection.automatic || candidate.host.id === selection.hostId)
|
||||||
|
&& !!candidate.host.inventory.runtimeImages?.includes(selection.image ?? '')
|
||||||
|
&& !!candidate.host.inventory.libraries?.some(library => library.id === selection.libraryId && ['ready', 'outdated'].includes(library.state));
|
||||||
|
const relevant = candidates.filter(candidate => selections.some(selection => eligible(candidate, selection)));
|
||||||
|
const planned = new Map<string, number>();
|
||||||
|
let estimatedAvailable = 0;
|
||||||
|
for (const selection of selections) {
|
||||||
|
const candidate = relevant.find(candidate => {
|
||||||
|
const used = planned.get(candidate.host.id) ?? 0;
|
||||||
|
return eligible(candidate, selection) && freePorts(candidate, selection.network?.bindAddress ?? selection.bindAddress) > used
|
||||||
|
&& candidate.freeCpu !== null && candidate.freeMemory !== null
|
||||||
|
&& candidate.freeCpu >= (used + 1) * candidate.runningCpu
|
||||||
|
&& candidate.freeMemory >= (used + 1) * candidate.runningMemory;
|
||||||
|
});
|
||||||
|
if (candidate) { estimatedAvailable++; planned.set(candidate.host.id, (planned.get(candidate.host.id) ?? 0) + 1); }
|
||||||
|
}
|
||||||
|
const warnings: string[] = [];
|
||||||
|
const unknown = relevant.some(candidate => candidate.freeCpu === null || candidate.freeMemory === null);
|
||||||
|
if (selections.length && !relevant.length) warnings.push('Waiting for an eligible host with the selected runtime and game files. The launch can be queued now.');
|
||||||
|
else if (unknown) warnings.push('Some host CPU or memory measurements are unavailable. Provisioning waits for fresh capacity measurements.');
|
||||||
|
else if (estimatedAvailable < selections.length) warnings.push(`Current capacity is estimated for ${estimatedAvailable} of ${selections.length} servers. Remaining launches wait for capacity, including any removals in this collection.`);
|
||||||
|
const startupSlots = Math.max(0, snapshot.startupLimit - snapshot.active);
|
||||||
|
if (selections.length && !startupSlots) warnings.push('All startup slots are in use. New launches will wait in the queue.');
|
||||||
|
if (selections.length && snapshot.queued) warnings.push(`${snapshot.queued} server${snapshot.queued === 1 ? ' is' : 's are'} already queued.`);
|
||||||
|
return { availability: { updatedAt: snapshot.updatedAt, requested: selections.length, estimatedAvailable: unknown ? null : estimatedAvailable,
|
||||||
|
portsAvailable: relevant.reduce((total, candidate) => total + freePorts(candidate), 0),
|
||||||
|
cpuCoresAvailable: unknown ? null : relevant.reduce((total, candidate) => total + (candidate.freeCpu ?? 0), 0),
|
||||||
|
memoryBytesAvailable: unknown ? null : relevant.reduce((total, candidate) => total + (candidate.freeMemory ?? 0), 0),
|
||||||
|
startupSlots, startupLimit: snapshot.startupLimit, queued: snapshot.queued }, warnings };
|
||||||
|
}
|
||||||
@@ -5,6 +5,7 @@ import { isIP } from 'node:net';
|
|||||||
import { db } from '../db/index.js';
|
import { db } from '../db/index.js';
|
||||||
import { ensureHostSchema } from './schema.js';
|
import { ensureHostSchema } from './schema.js';
|
||||||
import { recordInstanceResources, validateInstanceResources } from './calibration.js';
|
import { recordInstanceResources, validateInstanceResources } from './calibration.js';
|
||||||
|
import { refreshHostAvailability, scheduleAvailabilityRefresh } from './availability.js';
|
||||||
import type { Host, HostDecommission, HostDeletion, HostInventory, HostMode, HostOperation, HostOperationState } from '../../hosts.js';
|
import type { Host, HostDecommission, HostDeletion, HostInventory, HostMode, HostOperation, HostOperationState } from '../../hosts.js';
|
||||||
|
|
||||||
export const AGENT_PROTOCOL_VERSION = 1;
|
export const AGENT_PROTOCOL_VERSION = 1;
|
||||||
@@ -83,6 +84,17 @@ export function hostDeletionStatus(id: string): HostDeletion {
|
|||||||
const names = instances.slice(0, 5).map(instance => instance.server).join(', ') + (instances.length > 5 ? ', …' : '');
|
const names = instances.slice(0, 5).map(instance => instance.server).join(', ') + (instances.length > 5 ? ', …' : '');
|
||||||
blockers.push(`Remove the ${instances.length} server instance${instances.length === 1 ? '' : 's'} assigned to this host first (${names}). Queued and failed deployments also require removal.`);
|
blockers.push(`Remove the ${instances.length} server instance${instances.length === 1 ? '' : 's'} assigned to this host first (${names}). Queued and failed deployments also require removal.`);
|
||||||
}
|
}
|
||||||
|
const pinned = handle.prepare(`SELECT d.server FROM server_deployments d
|
||||||
|
JOIN native_creation_intents q ON q.deployment_id=d.id
|
||||||
|
LEFT JOIN native_provisioning n ON n.deployment_id=d.id
|
||||||
|
WHERE d.action='create' AND d.server_deleted_at IS NULL
|
||||||
|
AND d.state IN ('queued','waiting','running','awaiting-health','interrupted') AND n.deployment_id IS NULL
|
||||||
|
AND json_extract(q.selection,'$.hostId')=? AND COALESCE(json_extract(q.selection,'$.automatic'),0)!=1
|
||||||
|
ORDER BY d.server LIMIT 6`).all(id) as { server: string }[];
|
||||||
|
if (pinned.length) {
|
||||||
|
const names = pinned.slice(0, 5).map(job => job.server).join(', ') + (pinned.length > 5 ? ', …' : '');
|
||||||
|
blockers.push(`Cancel the queued server launches pinned to this host before deleting it (${names}).`);
|
||||||
|
}
|
||||||
const ports = handle.prepare('SELECT COUNT(*) AS count FROM ec_host_ports WHERE host_id=?').get(id) as { count: number };
|
const ports = handle.prepare('SELECT COUNT(*) AS count FROM ec_host_ports WHERE host_id=?').get(id) as { count: number };
|
||||||
if (ports.count && !instances.length) blockers.push(`This host still has ${ports.count} reserved game port${ports.count === 1 ? '' : 's'}. Finish or reconcile the corresponding server removals first.`);
|
if (ports.count && !instances.length) blockers.push(`This host still has ${ports.count} reserved game port${ports.count === 1 ? '' : 's'}. Finish or reconcile the corresponding server removals first.`);
|
||||||
// A pending command may have reached the agent before its acknowledgement.
|
// A pending command may have reached the agent before its acknowledgement.
|
||||||
@@ -174,6 +186,7 @@ export function createHost(input: Record<string, unknown>): Host {
|
|||||||
const bind = input.bindAddress === undefined ? '0.0.0.0' : textField(input.bindAddress, 'Bind address');
|
const bind = input.bindAddress === undefined ? '0.0.0.0' : textField(input.bindAddress, 'Bind address');
|
||||||
if (!isIP(bind)) throw new Error('Bind address must be an IPv4 or IPv6 address.');
|
if (!isIP(bind)) throw new Error('Bind address must be an IPv4 or IPv6 address.');
|
||||||
database().prepare('INSERT INTO ec_hosts(id,name,port_min,port_max,bind_address,created_at) VALUES(?,?,?,?,?,?)').run(id, name, min, max, bind, now());
|
database().prepare('INSERT INTO ec_hosts(id,name,port_min,port_max,bind_address,created_at) VALUES(?,?,?,?,?,?)').run(id, name, min, max, bind, now());
|
||||||
|
scheduleAvailabilityRefresh();
|
||||||
return requireHost(id);
|
return requireHost(id);
|
||||||
}
|
}
|
||||||
export function updateHost(id: string, input: Record<string, unknown>): Host {
|
export function updateHost(id: string, input: Record<string, unknown>): Host {
|
||||||
@@ -190,6 +203,7 @@ export function updateHost(id: string, input: Record<string, unknown>): Host {
|
|||||||
const minimum = input.minimumFreeBytes ?? host.minimumFreeBytes;
|
const minimum = input.minimumFreeBytes ?? host.minimumFreeBytes;
|
||||||
if (typeof minimum !== 'number' || !Number.isSafeInteger(minimum) || minimum < 0) throw new Error('Minimum free storage must be a nonnegative byte count.');
|
if (typeof minimum !== 'number' || !Number.isSafeInteger(minimum) || minimum < 0) throw new Error('Minimum free storage must be a nonnegative byte count.');
|
||||||
database().prepare('UPDATE ec_hosts SET name=?,mode=?,port_min=?,port_max=?,bind_address=?,public_address=?,minimum_free_bytes=? WHERE id=?').run(name, mode, min, max, bind, publicAddress, minimum, id);
|
database().prepare('UPDATE ec_hosts SET name=?,mode=?,port_min=?,port_max=?,bind_address=?,public_address=?,minimum_free_bytes=? WHERE id=?').run(name, mode, min, max, bind, publicAddress, minimum, id);
|
||||||
|
scheduleAvailabilityRefresh();
|
||||||
return requireHost(id);
|
return requireHost(id);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -256,6 +270,7 @@ export function enrollHost(token: unknown, rawInventory: unknown): { hostId: str
|
|||||||
const credential = issueCredential(enrollment.host_id);
|
const credential = issueCredential(enrollment.host_id);
|
||||||
database().prepare('UPDATE ec_hosts SET enrolled_at=?,last_heartbeat=?,hostname=?,inventory=? WHERE id=?').run(now(), now(), inventory.hostname ?? '', JSON.stringify(inventory), enrollment.host_id);
|
database().prepare('UPDATE ec_hosts SET enrolled_at=?,last_heartbeat=?,hostname=?,inventory=? WHERE id=?').run(now(), now(), inventory.hostname ?? '', JSON.stringify(inventory), enrollment.host_id);
|
||||||
recordResources(enrollment.host_id, inventory, now());
|
recordResources(enrollment.host_id, inventory, now());
|
||||||
|
scheduleAvailabilityRefresh();
|
||||||
return { hostId: enrollment.host_id, credential, heartbeatSeconds: 15 };
|
return { hostId: enrollment.host_id, credential, heartbeatSeconds: 15 };
|
||||||
})();
|
})();
|
||||||
}
|
}
|
||||||
@@ -270,6 +285,7 @@ export function revokeCredentials(hostId: string): void {
|
|||||||
const host = requireHost(hostId);
|
const host = requireHost(hostId);
|
||||||
if (host.decommission && !host.decommission.retryable) throw new Error('Host cleanup needs its credentials until removal is confirmed.');
|
if (host.decommission && !host.decommission.retryable) throw new Error('Host cleanup needs its credentials until removal is confirmed.');
|
||||||
database().prepare('UPDATE ec_host_credentials SET revoked_at=? WHERE host_id=? AND revoked_at IS NULL').run(now(), hostId);
|
database().prepare('UPDATE ec_host_credentials SET revoked_at=? WHERE host_id=? AND revoked_at IS NULL').run(now(), hostId);
|
||||||
|
scheduleAvailabilityRefresh();
|
||||||
}
|
}
|
||||||
export function rotateCredential(hostId: string, nextCredential?: string): string {
|
export function rotateCredential(hostId: string, nextCredential?: string): string {
|
||||||
requireHost(hostId);
|
requireHost(hostId);
|
||||||
@@ -292,7 +308,9 @@ export function heartbeat(hostId: string, value: unknown): Host {
|
|||||||
database().prepare('UPDATE ec_hosts SET last_heartbeat=?,hostname=?,inventory=? WHERE id=?').run(timestamp, inventory.hostname ?? '', JSON.stringify(inventory), hostId);
|
database().prepare('UPDATE ec_hosts SET last_heartbeat=?,hostname=?,inventory=? WHERE id=?').run(timestamp, inventory.hostname ?? '', JSON.stringify(inventory), hostId);
|
||||||
recordResources(hostId, inventory, timestamp);
|
recordResources(hostId, inventory, timestamp);
|
||||||
})();
|
})();
|
||||||
return requireHost(hostId);
|
const host = requireHost(hostId);
|
||||||
|
refreshHostAvailability(host);
|
||||||
|
return host;
|
||||||
}
|
}
|
||||||
|
|
||||||
function recordResources(hostId: string, inventory: HostInventory, timestamp: string): void {
|
function recordResources(hostId: string, inventory: HostInventory, timestamp: string): void {
|
||||||
@@ -504,7 +522,11 @@ export function reserveHostPort(hostId: string, instanceId: string, options: { p
|
|||||||
const allocated = options.port ?? nextHostPort(hostId, [], bind, protocol);
|
const allocated = options.port ?? nextHostPort(hostId, [], bind, protocol);
|
||||||
if (allocated === null || !hostPortAvailable(hostId, allocated, bind, protocol, instanceId)) throw new Error('No available game port on the selected host.');
|
if (allocated === null || !hostPortAvailable(hostId, allocated, bind, protocol, instanceId)) throw new Error('No available game port on the selected host.');
|
||||||
database().prepare('INSERT INTO ec_host_ports(host_id,instance_id,bind_address,protocol,port,created_at) VALUES(?,?,?,?,?,?)').run(hostId, instanceId, bind, protocol, allocated, now());
|
database().prepare('INSERT INTO ec_host_ports(host_id,instance_id,bind_address,protocol,port,created_at) VALUES(?,?,?,?,?,?)').run(hostId, instanceId, bind, protocol, allocated, now());
|
||||||
|
scheduleAvailabilityRefresh();
|
||||||
return allocated;
|
return allocated;
|
||||||
})();
|
})();
|
||||||
}
|
}
|
||||||
export function releaseHostPorts(instanceId: string): void { database().prepare('DELETE FROM ec_host_ports WHERE instance_id=?').run(instanceId); }
|
export function releaseHostPorts(instanceId: string): void {
|
||||||
|
database().prepare('DELETE FROM ec_host_ports WHERE instance_id=?').run(instanceId);
|
||||||
|
scheduleAvailabilityRefresh();
|
||||||
|
}
|
||||||
|
|||||||
@@ -2,12 +2,46 @@ import { db } from '../db/index.js';
|
|||||||
import { config } from '../config.js';
|
import { config } from '../config.js';
|
||||||
import { AGENT_PROTOCOL_VERSION, getHost, queueHostOperation } from './index.js';
|
import { AGENT_PROTOCOL_VERSION, getHost, queueHostOperation } from './index.js';
|
||||||
import { getHostLibraryCache, latestPublishedLibraries, queueLibraryDownload } from '../game-files/index.js';
|
import { getHostLibraryCache, latestPublishedLibraries, queueLibraryDownload } from '../game-files/index.js';
|
||||||
|
import { GAME_FILE_GAMES } from '../../game-files.js';
|
||||||
import type { Host, HostOperationState, HostPreparation, HostPreparationItem } from '../../hosts.js';
|
import type { Host, HostOperationState, HostPreparation, HostPreparationItem } from '../../hosts.js';
|
||||||
|
|
||||||
const activeStates = ['pending', 'accepted', 'running', 'unknown'];
|
const activeStates = ['pending', 'accepted', 'running', 'unknown'];
|
||||||
const supersededDownload = 'Superseded by newer game files before delivery to the host.';
|
const supersededDownload = 'Superseded by newer game files before delivery to the host.';
|
||||||
type PreparationOperation = { id: string; state: HostOperationState; updated_at: string; error: string | null };
|
type PreparationOperation = { id: string; state: HostOperationState; updated_at: string; error: string | null };
|
||||||
type Target = { item: HostPreparationItem; image?: string; versionId?: string; requiredBytes: number; operation?: PreparationOperation };
|
type Target = { item: HostPreparationItem; image?: string; versionId?: string; requiredBytes: number; operation?: PreparationOperation };
|
||||||
|
type QueuedSelection = { libraryId: string; image: string | null };
|
||||||
|
type LibraryTarget = Pick<ReturnType<typeof latestPublishedLibraries>[number], 'id' | 'sizeBytes' | 'manifestSha256'> & { name: string; queued?: boolean };
|
||||||
|
|
||||||
|
/** Accepted requests retain their prerequisites until placement owns an instance. */
|
||||||
|
function queuedSelections(hostId: string): QueuedSelection[] {
|
||||||
|
// Standalone host/library readers can initialize before provisioning migrations.
|
||||||
|
if (!db().prepare("SELECT 1 FROM sqlite_master WHERE type='table' AND name='native_creation_intents'").get()) return [];
|
||||||
|
return db().prepare(`SELECT DISTINCT json_extract(q.selection,'$.libraryId') AS libraryId,
|
||||||
|
json_extract(q.selection,'$.image') AS image FROM native_creation_intents q
|
||||||
|
JOIN server_deployments d ON d.id=q.deployment_id
|
||||||
|
LEFT JOIN native_provisioning n ON n.deployment_id=d.id
|
||||||
|
WHERE d.action='create' AND d.server_deleted_at IS NULL AND n.deployment_id IS NULL
|
||||||
|
AND d.state IN ('queued','waiting','running','awaiting-health','interrupted')
|
||||||
|
AND (json_extract(q.selection,'$.automatic')=1 OR json_extract(q.selection,'$.hostId') IS NULL
|
||||||
|
OR json_extract(q.selection,'$.hostId')=?)`).all(hostId) as QueuedSelection[];
|
||||||
|
}
|
||||||
|
|
||||||
|
function targetLibraries(selections: QueuedSelection[]): LibraryTarget[] {
|
||||||
|
const latest = latestPublishedLibraries();
|
||||||
|
const latestIds = new Set(latest.map(version => version.id));
|
||||||
|
const queued = new Map<string, LibraryTarget>();
|
||||||
|
const get = db().prepare(`SELECT v.id,l.game,v.version,v.size_bytes AS sizeBytes,v.manifest_sha256 AS manifestSha256
|
||||||
|
FROM game_file_versions v JOIN game_file_libraries l ON l.id=v.library_id
|
||||||
|
WHERE v.id=? AND v.state='ready' AND NOT EXISTS(SELECT 1 FROM game_file_cleanup c WHERE c.version_id=v.id)`);
|
||||||
|
for (const selection of selections) {
|
||||||
|
if (latestIds.has(selection.libraryId) || queued.has(selection.libraryId)) continue;
|
||||||
|
const version = get.get(selection.libraryId) as { id: string; game: string; version: string; sizeBytes: number; manifestSha256: string } | undefined;
|
||||||
|
if (version) queued.set(version.id, { ...version, queued: true,
|
||||||
|
name: `${GAME_FILE_GAMES.find(game => game.id === version.game)?.name ?? version.game} · Queued version ${version.version}` });
|
||||||
|
}
|
||||||
|
// Finish prerequisites for accepted work before downloading newer optional files.
|
||||||
|
return [...queued.values(), ...latest];
|
||||||
|
}
|
||||||
|
|
||||||
function latestOperation(hostId: string, kind: string, field: string, value: string): PreparationOperation | undefined {
|
function latestOperation(hostId: string, kind: string, field: string, value: string): PreparationOperation | undefined {
|
||||||
return db().prepare(`SELECT id,state,updated_at,error FROM ec_host_operations
|
return db().prepare(`SELECT id,state,updated_at,error FROM ec_host_operations
|
||||||
@@ -43,19 +77,24 @@ function hostBlocker(host: Host): { state: 'waiting' | 'blocked'; detail: string
|
|||||||
}
|
}
|
||||||
|
|
||||||
function targets(host: Host): Target[] {
|
function targets(host: Host): Target[] {
|
||||||
const image = config.komodo.defaultImage;
|
const selections = queuedSelections(host.id);
|
||||||
|
const images = new Set([config.komodo.defaultImage, ...selections.flatMap(selection => selection.image ? [selection.image] : [])]);
|
||||||
|
const result: Target[] = [...images].map(image => {
|
||||||
const runtimeReady = host.inventory.runtimeImages?.includes(image);
|
const runtimeReady = host.inventory.runtimeImages?.includes(image);
|
||||||
const result: Target[] = [{ image, requiredBytes: host.minimumFreeBytes,
|
return { image, requiredBytes: host.minimumFreeBytes,
|
||||||
operation: latestOperation(host.id, 'runtime.pull', 'image', image),
|
operation: latestOperation(host.id, 'runtime.pull', 'image', image),
|
||||||
item: { kind: 'runtime', label: image, state: runtimeReady ? 'ready' : 'waiting', detail: runtimeReady ? 'Runtime is cached.' : 'Runtime will be downloaded automatically.' } }];
|
item: { kind: 'runtime', label: image, state: runtimeReady ? 'ready' : 'waiting', detail: runtimeReady ? 'Runtime is cached.' : 'Runtime will be downloaded automatically.' } };
|
||||||
for (const version of latestPublishedLibraries()) {
|
});
|
||||||
|
for (const version of targetLibraries(selections)) {
|
||||||
const cache = getHostLibraryCache(host.id, version.id);
|
const cache = getHostLibraryCache(host.id, version.id);
|
||||||
const matches = cache?.manifestSha256 === version.manifestSha256;
|
const matches = cache?.manifestSha256 === version.manifestSha256;
|
||||||
const ready = matches && ['ready', 'outdated'].includes(cache!.status) && cache!.bytesDownloaded === version.sizeBytes;
|
const ready = matches && ['ready', 'outdated'].includes(cache!.status) && cache!.bytesDownloaded === version.sizeBytes;
|
||||||
result.push({ versionId: version.id, requiredBytes: version.sizeBytes - (matches ? cache!.bytesDownloaded : 0) + host.minimumFreeBytes,
|
result.push({ versionId: version.id, requiredBytes: version.sizeBytes - (matches ? cache!.bytesDownloaded : 0) + host.minimumFreeBytes,
|
||||||
operation: latestOperation(host.id, 'library.download', 'libraryVersionId', version.id),
|
operation: latestOperation(host.id, 'library.download', 'libraryVersionId', version.id),
|
||||||
item: { kind: 'library', label: version.name, state: ready ? 'ready' : 'waiting',
|
item: { kind: 'library', label: version.name, state: ready ? 'ready' : 'waiting',
|
||||||
detail: ready ? 'The latest game files are downloaded and verified.' : 'The latest game files will be downloaded and verified automatically.',
|
detail: version.queued
|
||||||
|
? ready ? 'Game files selected by queued launches are downloaded and verified.' : 'Game files selected by queued launches will be downloaded and verified automatically.'
|
||||||
|
: ready ? 'The latest game files are downloaded and verified.' : 'The latest game files will be downloaded and verified automatically.',
|
||||||
bytesDownloaded: cache?.bytesDownloaded ?? 0, bytesTotal: version.sizeBytes, ...(cache ? { cache } : {}) } });
|
bytesDownloaded: cache?.bytesDownloaded ?? 0, bytesTotal: version.sizeBytes, ...(cache ? { cache } : {}) } });
|
||||||
}
|
}
|
||||||
const blocker = hostBlocker(host);
|
const blocker = hostBlocker(host);
|
||||||
@@ -100,13 +139,13 @@ export function presentHostPreparation(host: Host): Host {
|
|||||||
}
|
}
|
||||||
|
|
||||||
function supersedeUndeliveredDownloads(hostId: string): void {
|
function supersedeUndeliveredDownloads(hostId: string): void {
|
||||||
const latest = new Set(latestPublishedLibraries().map(version => version.id));
|
const required = new Set(targetLibraries(queuedSelections(hostId)).map(version => version.id));
|
||||||
const pending = db().prepare(`SELECT id,json_extract(payload,'$.libraryVersionId') AS versionId
|
const pending = db().prepare(`SELECT id,json_extract(payload,'$.libraryVersionId') AS versionId
|
||||||
FROM ec_host_operations WHERE host_id=? AND kind='library.download' AND state='pending'
|
FROM ec_host_operations WHERE host_id=? AND kind='library.download' AND state='pending'
|
||||||
AND last_delivered_at IS NULL AND json_extract(payload,'$.automatic')=1`).all(hostId) as { id: string; versionId: string }[];
|
AND last_delivered_at IS NULL AND json_extract(payload,'$.automatic')=1`).all(hostId) as { id: string; versionId: string }[];
|
||||||
const now = new Date().toISOString();
|
const now = new Date().toISOString();
|
||||||
for (const operation of pending) {
|
for (const operation of pending) {
|
||||||
if (latest.has(operation.versionId)) continue;
|
if (required.has(operation.versionId)) continue;
|
||||||
// Delivered operations may already be writing files. Only unsubmitted
|
// Delivered operations may already be writing files. Only unsubmitted
|
||||||
// automatic work can be replaced without consulting the agent journal.
|
// automatic work can be replaced without consulting the agent journal.
|
||||||
db().prepare("UPDATE ec_host_operations SET state='failed',error=?,updated_at=? WHERE id=?").run(supersededDownload, now, operation.id);
|
db().prepare("UPDATE ec_host_operations SET state='failed',error=?,updated_at=? WHERE id=?").run(supersededDownload, now, operation.id);
|
||||||
|
|||||||
@@ -3,6 +3,8 @@ import type Database from 'better-sqlite3';
|
|||||||
const initialized = new WeakSet<Database.Database>();
|
const initialized = new WeakSet<Database.Database>();
|
||||||
export function ensureHostSchema(handle: Database.Database): void {
|
export function ensureHostSchema(handle: Database.Database): void {
|
||||||
if (initialized.has(handle)) return;
|
if (initialized.has(handle)) return;
|
||||||
|
// Background readers open only after controller migrations have completed.
|
||||||
|
if (handle.readonly) { initialized.add(handle); return; }
|
||||||
handle.exec(`
|
handle.exec(`
|
||||||
CREATE TABLE IF NOT EXISTS ec_hosts (
|
CREATE TABLE IF NOT EXISTS ec_hosts (
|
||||||
id TEXT PRIMARY KEY, name TEXT NOT NULL, hostname TEXT NOT NULL DEFAULT '',
|
id TEXT PRIMARY KEY, name TEXT NOT NULL, hostname TEXT NOT NULL DEFAULT '',
|
||||||
|
|||||||
@@ -6,9 +6,13 @@ import { recordProgress } from './progress.js';
|
|||||||
|
|
||||||
export function canCancelCreation(id: string): boolean {
|
export function canCancelCreation(id: string): boolean {
|
||||||
return !!db().prepare(`SELECT 1 FROM server_deployments d
|
return !!db().prepare(`SELECT 1 FROM server_deployments d
|
||||||
JOIN native_provisioning n ON n.deployment_id=d.id
|
LEFT JOIN native_provisioning n ON n.deployment_id=d.id
|
||||||
JOIN native_instances i ON i.id=n.instance_id AND i.generation=n.generation AND i.retired_at IS NULL
|
LEFT JOIN native_instances i ON i.id=n.instance_id AND i.generation=n.generation AND i.retired_at IS NULL
|
||||||
WHERE d.id=? AND d.action='create' AND d.server_deleted_at IS NULL
|
WHERE d.id=? AND d.action='create' AND d.server_deleted_at IS NULL
|
||||||
|
AND (i.id IS NOT NULL OR (n.deployment_id IS NULL
|
||||||
|
AND EXISTS (SELECT 1 FROM native_creation_intents intent WHERE intent.deployment_id=d.id)
|
||||||
|
AND NOT EXISTS (SELECT 1 FROM native_instances replacement WHERE replacement.server=d.server AND replacement.retired_at IS NULL)
|
||||||
|
AND NOT EXISTS (SELECT 1 FROM servers replacement WHERE replacement.name=d.server AND replacement.destroyed_at IS NULL)))
|
||||||
AND d.state IN ('queued','waiting','running','awaiting-health','interrupted','failed')
|
AND d.state IN ('queued','waiting','running','awaiting-health','interrupted','failed')
|
||||||
AND NOT EXISTS (SELECT 1 FROM server_deployments other WHERE other.server=d.server
|
AND NOT EXISTS (SELECT 1 FROM server_deployments other WHERE other.server=d.server
|
||||||
AND other.server_deleted_at IS NULL AND other.rowid>d.rowid AND other.state!='cancelled'
|
AND other.server_deleted_at IS NULL AND other.rowid>d.rowid AND other.state!='cancelled'
|
||||||
@@ -23,9 +27,9 @@ export function cancelCreation(id: string, triggeredBy = 'ui'): string | null {
|
|||||||
if (prior) return prior.cleanup_id;
|
if (prior) return prior.cleanup_id;
|
||||||
if (!canCancelCreation(id)) return null;
|
if (!canCancelCreation(id)) return null;
|
||||||
const job = db().prepare(`SELECT d.server,n.operation_id,n.stage_operation_id FROM server_deployments d
|
const job = db().prepare(`SELECT d.server,n.operation_id,n.stage_operation_id FROM server_deployments d
|
||||||
JOIN native_provisioning n ON n.deployment_id=d.id WHERE d.id=?`).get(id) as
|
LEFT JOIN native_provisioning n ON n.deployment_id=d.id WHERE d.id=?`).get(id) as
|
||||||
{ server: string; operation_id: string | null; stage_operation_id: string | null };
|
{ server: string; operation_id: string | null; stage_operation_id: string | null };
|
||||||
const instance = nativeInstance(job.server)!;
|
const instance = nativeInstance(job.server);
|
||||||
// Replace a removal still waiting on occupancy, including one whose asynchronous probe is outstanding.
|
// Replace a removal still waiting on occupancy, including one whose asynchronous probe is outstanding.
|
||||||
for (const removal of db().prepare("SELECT id FROM server_deployments WHERE server=? AND action='destroy' AND state='waiting' AND server_deleted_at IS NULL").all(job.server) as { id: string }[]) {
|
for (const removal of db().prepare("SELECT id FROM server_deployments WHERE server=? AND action='destroy' AND state='waiting' AND server_deleted_at IS NULL").all(job.server) as { id: string }[]) {
|
||||||
cancelRemoval(removal.id);
|
cancelRemoval(removal.id);
|
||||||
@@ -37,13 +41,16 @@ export function cancelCreation(id: string, triggeredBy = 'ui'): string | null {
|
|||||||
event: { kind: 'cancelled', message: 'Creation cancelled. Its partial instance will be removed without a player-count check.', details: { cleanupId } } });
|
event: { kind: 'cancelled', message: 'Creation cancelled. Its partial instance will be removed without a player-count check.', details: { cleanupId } } });
|
||||||
if (!job.operation_id && !job.stage_operation_id) {
|
if (!job.operation_id && !job.stage_operation_id) {
|
||||||
// No remote mutation was ever queued. The reservation is the only resource to remove.
|
// No remote mutation was ever queued. The reservation is the only resource to remove.
|
||||||
|
if (instance) {
|
||||||
db().prepare('UPDATE native_instances SET retired_at=? WHERE id=?').run(new Date().toISOString(), instance.id);
|
db().prepare('UPDATE native_instances SET retired_at=? WHERE id=?').run(new Date().toISOString(), instance.id);
|
||||||
releaseHostPorts(instance.id);
|
releaseHostPorts(instance.id);
|
||||||
|
}
|
||||||
deploymentStage(cleanupId, 'Unsubmitted creation cancelled; reservations released', { phase: 'cleanup',
|
deploymentStage(cleanupId, 'Unsubmitted creation cancelled; reservations released', { phase: 'cleanup',
|
||||||
checkpoint: { key: 'cleanup', label: 'No host command submitted; local reservations released' } });
|
checkpoint: { key: 'cleanup', label: 'No host command submitted; local reservations released' } });
|
||||||
finishDeployment(cleanupId, { ok: true });
|
finishDeployment(cleanupId, { ok: true });
|
||||||
finishCreationCancellation(cleanupId);
|
finishCreationCancellation(cleanupId);
|
||||||
} else {
|
} else {
|
||||||
|
if (!instance) throw new Error('The original native instance is unavailable.');
|
||||||
const operation = queueHostOperation(instance.host_id, 'instance.cancel-create', { createOperationId: job.operation_id ?? id },
|
const operation = queueHostOperation(instance.host_id, 'instance.cancel-create', { createOperationId: job.operation_id ?? id },
|
||||||
{ instanceId: instance.id, generation: instance.generation, idempotencyKey: cleanupId });
|
{ instanceId: instance.id, generation: instance.generation, idempotencyKey: cleanupId });
|
||||||
db().prepare('UPDATE native_provisioning SET operation_id=? WHERE deployment_id=?').run(operation.id, cleanupId);
|
db().prepare('UPDATE native_provisioning SET operation_id=? WHERE deployment_id=?').run(operation.id, cleanupId);
|
||||||
@@ -57,12 +64,12 @@ export function cancelCreation(id: string, triggeredBy = 'ui'): string | null {
|
|||||||
|
|
||||||
export function finishCreationCancellation(cleanupId: string): void {
|
export function finishCreationCancellation(cleanupId: string): void {
|
||||||
const cancellation = db().prepare(`SELECT c.creation_id,n.operation_id FROM native_creation_cancellations c
|
const cancellation = db().prepare(`SELECT c.creation_id,n.operation_id FROM native_creation_cancellations c
|
||||||
JOIN native_provisioning n ON n.deployment_id=c.creation_id WHERE c.cleanup_id=?`).get(cleanupId) as
|
LEFT JOIN native_provisioning n ON n.deployment_id=c.creation_id WHERE c.cleanup_id=?`).get(cleanupId) as
|
||||||
{ creation_id: string; operation_id: string | null } | undefined;
|
{ creation_id: string; operation_id: string | null } | undefined;
|
||||||
if (!cancellation) return;
|
if (!cancellation) return;
|
||||||
const now = new Date().toISOString();
|
const now = new Date().toISOString();
|
||||||
db().prepare('UPDATE server_deployments SET server_deleted_at=? WHERE id IN (?,?)').run(now, cancellation.creation_id, cleanupId);
|
db().prepare('UPDATE server_deployments SET server_deleted_at=? WHERE id IN (?,?)').run(now, cancellation.creation_id, cleanupId);
|
||||||
deploymentStage(cancellation.creation_id, 'Creation cancelled; partial instance removed', { blocker: null,
|
deploymentStage(cancellation.creation_id, 'Creation cancelled; cleanup completed', { blocker: null,
|
||||||
checkpoint: { key: 'cleanup', phase: 'cleanup', label: 'Cancellation cleanup confirmed; instance and reservations removed' } });
|
checkpoint: { key: 'cleanup', phase: 'cleanup', label: 'Cancellation cleanup confirmed; instance and reservations removed' } });
|
||||||
// The original command may have lost its receipt. Cleanup proves that its effects are now gone.
|
// The original command may have lost its receipt. Cleanup proves that its effects are now gone.
|
||||||
if (cancellation.operation_id) {
|
if (cancellation.operation_id) {
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ import { readProgress, recordProgress, type ProgressUpdate } from './progress.js
|
|||||||
import { bindNativeOperation } from './native.js';
|
import { bindNativeOperation } from './native.js';
|
||||||
import { workerActivity } from '../pipeline/activity.js';
|
import { workerActivity } from '../pipeline/activity.js';
|
||||||
import { canCancelCreation } from './cancel-creation.js';
|
import { canCancelCreation } from './cancel-creation.js';
|
||||||
|
import { nativeDeployment } from '../backends/native.js';
|
||||||
|
|
||||||
export function beginDeployment(server: string, action: ProvisioningAction, triggeredBy = 'ui'): string {
|
export function beginDeployment(server: string, action: ProvisioningAction, triggeredBy = 'ui'): string {
|
||||||
const id = randomUUID();
|
const id = randomUUID();
|
||||||
@@ -103,7 +104,7 @@ export function getDeployment(id: string, options: { events?: 'all' | 'recent' }
|
|||||||
(ProvisioningDeployment & { observation_json: string | null; phase_key: string | null; phase_started_at: string | null; observed_at: string | null }) | undefined;
|
(ProvisioningDeployment & { observation_json: string | null; phase_key: string | null; phase_started_at: string | null; observed_at: string | null }) | undefined;
|
||||||
if (!row) return null;
|
if (!row) return null;
|
||||||
const generatedAt = new Date().toISOString();
|
const generatedAt = new Date().toISOString();
|
||||||
const native = db().prepare('SELECT instance_id FROM native_provisioning WHERE deployment_id=?').get(id);
|
const native = nativeDeployment(id);
|
||||||
const agentOperation = native ? db().prepare(`SELECT o.id, o.host_id AS hostId, o.state, o.updated_at AS updatedAt, o.created_at AS createdAt, o.error
|
const agentOperation = native ? db().prepare(`SELECT o.id, o.host_id AS hostId, o.state, o.updated_at AS updatedAt, o.created_at AS createdAt, o.error
|
||||||
FROM native_provisioning n JOIN ec_host_operations o ON o.id=n.operation_id WHERE n.deployment_id=?`).get(id) as ProvisioningDetail['agentOperation'] : null;
|
FROM native_provisioning n JOIN ec_host_operations o ON o.id=n.operation_id WHERE n.deployment_id=?`).get(id) as ProvisioningDetail['agentOperation'] : null;
|
||||||
const cancellation = db().prepare(`SELECT c.cleanup_id AS cleanupId,d.state,d.stage,d.error FROM native_creation_cancellations c
|
const cancellation = db().prepare(`SELECT c.cleanup_id AS cleanupId,d.state,d.stage,d.error FROM native_creation_cancellations c
|
||||||
@@ -254,10 +255,11 @@ export function listDeployments() {
|
|||||||
|
|
||||||
export function interruptDeployments(): void {
|
export function interruptDeployments(): void {
|
||||||
db().transaction(() => {
|
db().transaction(() => {
|
||||||
const running = db().prepare("SELECT id FROM server_deployments WHERE state = 'running' AND id NOT IN (SELECT deployment_id FROM native_provisioning)").all() as { id: string }[];
|
const native = 'SELECT deployment_id FROM native_provisioning UNION ALL SELECT deployment_id FROM native_creation_intents';
|
||||||
|
const running = db().prepare(`SELECT id FROM server_deployments WHERE state = 'running' AND id NOT IN (${native})`).all() as { id: string }[];
|
||||||
const finishedAt = new Date().toISOString();
|
const finishedAt = new Date().toISOString();
|
||||||
const error = 'The application restarted before this operation finished. Check the server before retrying.';
|
const error = 'The application restarted before this operation finished. Check the server before retrying.';
|
||||||
db().prepare(`UPDATE server_deployments SET state = 'interrupted', finished_at = ?, error = ? WHERE state = 'running' AND id NOT IN (SELECT deployment_id FROM native_provisioning)`).run(finishedAt, error);
|
db().prepare(`UPDATE server_deployments SET state = 'interrupted', finished_at = ?, error = ? WHERE state = 'running' AND id NOT IN (${native})`).run(finishedAt, error);
|
||||||
for (const { id } of running) recordProgress(id, { blocker: null, event: { kind: 'outcome',
|
for (const { id } of running) recordProgress(id, { blocker: null, event: { kind: 'outcome',
|
||||||
message: 'Interrupted by application restart. Check the server before retrying.', details: { state: 'interrupted', finishedAt, error } } });
|
message: 'Interrupted by application restart. Check the server before retrying.', details: { state: 'interrupted', finishedAt, error } } });
|
||||||
})();
|
})();
|
||||||
|
|||||||
@@ -44,7 +44,7 @@ export type CreateServerFailure =
|
|||||||
| 'db'; // shouldn't happen, but wraps INSERT failures
|
| 'db'; // shouldn't happen, but wraps INSERT failures
|
||||||
|
|
||||||
export type CreateServerResult =
|
export type CreateServerResult =
|
||||||
| { ok: true; name: string; port: number; deploymentId: string }
|
| { ok: true; name: string; port: number | null; deploymentId: string }
|
||||||
| { ok: false; kind: CreateServerFailure; message: string };
|
| { ok: false; kind: CreateServerFailure; message: string };
|
||||||
|
|
||||||
export type DestroyServerResult =
|
export type DestroyServerResult =
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import { config } from '../config.js';
|
|||||||
import { db } from '../db/index.js';
|
import { db } from '../db/index.js';
|
||||||
import type { NativeInstance } from '../backends/native.js';
|
import type { NativeInstance } from '../backends/native.js';
|
||||||
import type { LaunchRecipePlan } from '../fleet/launch-recipes.js';
|
import type { LaunchRecipePlan } from '../fleet/launch-recipes.js';
|
||||||
|
import type { PlacementOptions } from '../hosts/capacity.js';
|
||||||
|
|
||||||
export interface NativeLaunchInput {
|
export interface NativeLaunchInput {
|
||||||
game: 't5' | 't6';
|
game: 't5' | 't6';
|
||||||
@@ -25,7 +26,7 @@ export interface PreparedNativeLaunch {
|
|||||||
let worker: Worker | null = null;
|
let worker: Worker | null = null;
|
||||||
let sequence = 0;
|
let sequence = 0;
|
||||||
const pending = new Map<number, {
|
const pending = new Map<number, {
|
||||||
resolve: (value: PreparedNativeLaunch) => void;
|
resolve: (value: unknown) => void;
|
||||||
reject: (error: Error) => void;
|
reject: (error: Error) => void;
|
||||||
timer: NodeJS.Timeout;
|
timer: NodeJS.Timeout;
|
||||||
}>();
|
}>();
|
||||||
@@ -39,14 +40,14 @@ function stop(error: Error, current = worker): Promise<number> | undefined {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/** The reader and JSON encoder run off the controller's request thread. */
|
/** The reader and JSON encoder run off the controller's request thread. */
|
||||||
export function prepareNativeLaunch(input: NativeLaunchInput): Promise<PreparedNativeLaunch> {
|
function run<T>(kind: 'launch' | 'placement', input: NativeLaunchInput | PlacementOptions): Promise<T> {
|
||||||
db(); // Migrations must finish before the independent read-only connection opens.
|
db(); // Migrations must finish before the independent read-only connection opens.
|
||||||
if (!worker) {
|
if (!worker) {
|
||||||
const current = worker = new Worker(resolve(process.env.NODE_ENV === 'production'
|
const current = worker = new Worker(resolve(process.env.NODE_ENV === 'production'
|
||||||
? 'build/launch/worker.mjs' : '.svelte-kit/launch/worker.mjs'), {
|
? 'build/launch/worker.mjs' : '.svelte-kit/launch/worker.mjs'), {
|
||||||
workerData: { dbPath: resolve(config.dbPath) }
|
workerData: { dbPath: resolve(config.dbPath) }
|
||||||
});
|
});
|
||||||
current.on('message', ({ id, value, error }: { id: number; value: PreparedNativeLaunch; error?: string }) => {
|
current.on('message', ({ id, value, error }: { id: number; value: unknown; error?: string }) => {
|
||||||
if (current !== worker) return;
|
if (current !== worker) return;
|
||||||
const request = pending.get(id);
|
const request = pending.get(id);
|
||||||
if (!request) return;
|
if (!request) return;
|
||||||
@@ -59,13 +60,18 @@ export function prepareNativeLaunch(input: NativeLaunchInput): Promise<PreparedN
|
|||||||
}
|
}
|
||||||
worker.ref();
|
worker.ref();
|
||||||
const id = ++sequence;
|
const id = ++sequence;
|
||||||
return new Promise((resolve, reject) => {
|
return new Promise<T>((resolve, reject) => {
|
||||||
const timer = setTimeout(() => { void stop(new Error('Launch preparation timed out; preparation will retry.')); }, 60_000);
|
const timer = setTimeout(() => { void stop(new Error('Launch preparation timed out; preparation will retry.')); }, 60_000);
|
||||||
pending.set(id, { resolve, reject, timer });
|
pending.set(id, { resolve: value => resolve(value as T), reject, timer });
|
||||||
worker!.postMessage({ id, input });
|
worker!.postMessage({ id, kind, input });
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function prepareNativeLaunch(input: NativeLaunchInput): Promise<PreparedNativeLaunch> { return run('launch', input); }
|
||||||
|
export function evaluateNativePlacement(input: PlacementOptions): Promise<{ hostId: string; lastHeartbeat: string | null }> {
|
||||||
|
return run('placement', input);
|
||||||
|
}
|
||||||
|
|
||||||
export async function closeLaunchPreparation(): Promise<void> {
|
export async function closeLaunchPreparation(): Promise<void> {
|
||||||
await stop(new Error('Launch preparation stopped.'));
|
await stop(new Error('Launch preparation stopped.'));
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,9 +5,17 @@ import { relDestForKind } from '../config.js';
|
|||||||
import { captureRecipe, summarizeRecipe } from '../fleet/launch-recipes.js';
|
import { captureRecipe, summarizeRecipe } from '../fleet/launch-recipes.js';
|
||||||
import { sha256Buffer } from '../fleet/files.js';
|
import { sha256Buffer } from '../fleet/files.js';
|
||||||
import type { NativeLaunchInput } from './launch-preparation.js';
|
import type { NativeLaunchInput } from './launch-preparation.js';
|
||||||
|
import { choosePlacement, type PlacementOptions } from '../hosts/capacity.js';
|
||||||
|
|
||||||
parentPort!.on('message', ({ id, input }: { id: number; input: NativeLaunchInput }) => {
|
parentPort!.on('message', (message: { id: number; kind: 'launch'; input: NativeLaunchInput } | { id: number; kind: 'placement'; input: PlacementOptions }) => {
|
||||||
|
const { id } = message;
|
||||||
try {
|
try {
|
||||||
|
if (message.kind === 'placement') {
|
||||||
|
const decision = db().transaction(() => choosePlacement(message.input))();
|
||||||
|
parentPort!.postMessage({ id, value: { hostId: decision.host.id, lastHeartbeat: decision.host.lastHeartbeat } });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const input = message.input;
|
||||||
// All file versions and composed configuration belong to one snapshot.
|
// All file versions and composed configuration belong to one snapshot.
|
||||||
const recipe = db().transaction(() => ({ ...captureRecipe(input.game, input.mapSlug),
|
const recipe = db().transaction(() => ({ ...captureRecipe(input.game, input.mapSlug),
|
||||||
native: input.native, image: input.instance.image }))();
|
native: input.native, image: input.instance.image }))();
|
||||||
|
|||||||
@@ -1,15 +1,15 @@
|
|||||||
import { isIP } from 'node:net';
|
import { isIP } from 'node:net';
|
||||||
import { setImmediate as yieldToController } from 'node:timers/promises';
|
import { setImmediate as yieldToController } from 'node:timers/promises';
|
||||||
import { CapacityError, choosePlacement, availableHostPort, type PlacementOptions } from '../hosts/capacity.js';
|
import { choosePlacement, availableHostPort, type PlacementOptions } from '../hosts/capacity.js';
|
||||||
import { randomBytes, randomUUID } from 'node:crypto';
|
import { randomBytes, randomUUID } from 'node:crypto';
|
||||||
import { posix } from 'node:path';
|
import { posix } from 'node:path';
|
||||||
import { db } from '../db/index.js';
|
import { db } from '../db/index.js';
|
||||||
import { config, relDestForKind, serverDir } from '../config.js';
|
import { config, relDestForKind, serverDir } from '../config.js';
|
||||||
import { nativeInstance, nativeRequest, type NativeInstance } from '../backends/native.js';
|
import { nativeInstance, nativeRequest, unplacedNativeCreation, type NativeInstance } from '../backends/native.js';
|
||||||
import { getHost, getHostOperationStatus, listHosts, queueHostOperation, queueHostOperationSerialized, releaseHostPorts, reserveHostPort } from '../hosts/index.js';
|
import { getHost, getHostOperationStatus, listHosts, queueHostOperation, queueHostOperationSerialized, releaseHostPorts, reserveHostPort } from '../hosts/index.js';
|
||||||
import { newestReadyLibraryId, assertReadyLibrary } from '../game-files/index.js';
|
import { newestReadyLibraryId, assertReadyLibrary } from '../game-files/index.js';
|
||||||
import { blueprintRecipe, captureRecipePlan, recipeForDeployment, recipeStorage, summarizeRecipe, type LaunchRecipePlan } from '../fleet/launch-recipes.js';
|
import { blueprintRecipe, captureRecipePlan, recipeForDeployment, recipeStorage, summarizeRecipe, type LaunchRecipePlan } from '../fleet/launch-recipes.js';
|
||||||
import { prepareNativeLaunch, type PreparedNativeLaunch } from './launch-preparation.js';
|
import { prepareNativeLaunch, evaluateNativePlacement, type PreparedNativeLaunch } from './launch-preparation.js';
|
||||||
import { getServerKey, saveServerKey } from '../fleet/server-keys.js';
|
import { getServerKey, saveServerKey } from '../fleet/server-keys.js';
|
||||||
import { activeBetaRemovalBlocker, getServer } from '../fleet/servers.js';
|
import { activeBetaRemovalBlocker, getServer } from '../fleet/servers.js';
|
||||||
import { nextLobbyNumber, type CreateServerInput, type CreateServerResult } from './instances.js';
|
import { nextLobbyNumber, type CreateServerInput, type CreateServerResult } from './instances.js';
|
||||||
@@ -30,11 +30,12 @@ import { cancelCreation, finishCreationCancellation } from './cancel-creation.js
|
|||||||
|
|
||||||
export type NativeSelection = import('../../hosts.js').NativePlacement;
|
export type NativeSelection = import('../../hosts.js').NativePlacement;
|
||||||
type NativeRecipe = LaunchRecipePlan & { native?: NativeSelection };
|
type NativeRecipe = LaunchRecipePlan & { native?: NativeSelection };
|
||||||
interface Job { id: string; server: string; action: ProvisioningAction; state: string; submitted_at: string | null; stage: string;
|
interface Job { selection?: string | null; id: string; server: string; action: ProvisioningAction; state: string; submitted_at: string | null; stage: string;
|
||||||
instance_id: string; generation: number; stage_operation_id: string | null; operation_id: string | null; files_json: string | null; applied_at: string | null }
|
instance_id: string; generation: number; stage_operation_id: string | null; operation_id: string | null; files_json: string | null; applied_at: string | null }
|
||||||
interface FilePlan { id: number; path: string; contentBase64: string | null; sha256: string | null; checkBefore: boolean; expectedBeforeSha: string | null }
|
interface FilePlan { id: number; path: string; contentBase64: string | null; sha256: string | null; checkBefore: boolean; expectedBeforeSha: string | null }
|
||||||
|
|
||||||
export function nativeSelection(input: Record<string, unknown>, game: string, options: Pick<PlacementOptions, 'planned'> = {}): NativeSelection {
|
/** Validate a launch request without selecting a host or consuming capacity. */
|
||||||
|
export function nativeLaunchSelection(input: Record<string, unknown>, game: string): NativeSelection {
|
||||||
if (input.hostId !== undefined && typeof input.hostId !== 'string') throw new Error('Invalid host.');
|
if (input.hostId !== undefined && typeof input.hostId !== 'string') throw new Error('Invalid host.');
|
||||||
const libraryId = newestReadyLibraryId(game);
|
const libraryId = newestReadyLibraryId(game);
|
||||||
if (!libraryId) throw new Error(`Upload ${game.toUpperCase()} game files before launching a server.`);
|
if (!libraryId) throw new Error(`Upload ${game.toUpperCase()} game files before launching a server.`);
|
||||||
@@ -44,16 +45,29 @@ export function nativeSelection(input: Record<string, unknown>, game: string, op
|
|||||||
const image = input.image === undefined || input.image === '' ? config.komodo.defaultImage : input.image;
|
const image = input.image === undefined || input.image === '' ? config.komodo.defaultImage : input.image;
|
||||||
if (typeof image !== 'string' || !/^[a-zA-Z0-9][a-zA-Z0-9._/:@-]{0,499}$/.test(image)) throw new Error('Invalid runtime image.');
|
if (typeof image !== 'string' || !/^[a-zA-Z0-9][a-zA-Z0-9._/:@-]{0,499}$/.test(image)) throw new Error('Invalid runtime image.');
|
||||||
const automatic = input.automatic === true || !input.hostId;
|
const automatic = input.automatic === true || !input.hostId;
|
||||||
|
if (!automatic) {
|
||||||
|
const host = getHost(input.hostId as string);
|
||||||
|
if (!host) throw new Error('The selected host was not found.');
|
||||||
|
if (host.decommission) throw new Error('The selected host is being removed. Choose another host.');
|
||||||
|
}
|
||||||
const rawNetwork = input.automatic === true ? input.network ?? {} : input;
|
const rawNetwork = input.automatic === true ? input.network ?? {} : input;
|
||||||
if (!rawNetwork || typeof rawNetwork !== 'object' || Array.isArray(rawNetwork)) throw new Error('Invalid network overrides.');
|
if (!rawNetwork || typeof rawNetwork !== 'object' || Array.isArray(rawNetwork)) throw new Error('Invalid network overrides.');
|
||||||
const network = rawNetwork as NativeSelection;
|
const network = rawNetwork as NativeSelection;
|
||||||
const bindAddress = network.bindAddress || undefined;
|
const bindAddress = network.bindAddress || undefined;
|
||||||
if (bindAddress !== undefined && (typeof bindAddress !== 'string' || !isIP(bindAddress))) throw new Error('Invalid bind address.');
|
if (bindAddress !== undefined && (typeof bindAddress !== 'string' || !isIP(bindAddress))) throw new Error('Invalid bind address.');
|
||||||
const { host } = choosePlacement({ hostId: automatic ? undefined : input.hostId as string, libraryId, image, bindAddress, ...options });
|
|
||||||
if (network.publicAddress !== undefined && (typeof network.publicAddress !== 'string' || network.publicAddress.length > 253)) throw new Error('Invalid public address.');
|
if (network.publicAddress !== undefined && (typeof network.publicAddress !== 'string' || network.publicAddress.length > 253)) throw new Error('Invalid public address.');
|
||||||
if (network.publicPort !== undefined && (!Number.isInteger(network.publicPort) || Number(network.publicPort) < 1 || Number(network.publicPort) > 65535)) throw new Error('Invalid public port.');
|
if (network.publicPort !== undefined && (!Number.isInteger(network.publicPort) || Number(network.publicPort) < 1 || Number(network.publicPort) > 65535)) throw new Error('Invalid public port.');
|
||||||
return { hostId: host.id, libraryId, image, bindAddress: bindAddress ?? host.bindAddress, publicAddress: network.publicAddress ?? host.publicAddress,
|
return { hostId: automatic ? undefined : input.hostId as string, libraryId, image, automatic,
|
||||||
publicPort: network.publicPort, automatic, network: { bindAddress, publicAddress: network.publicAddress, publicPort: network.publicPort } };
|
bindAddress, publicAddress: network.publicAddress, publicPort: network.publicPort,
|
||||||
|
network: { bindAddress, publicAddress: network.publicAddress, publicPort: network.publicPort } };
|
||||||
|
}
|
||||||
|
|
||||||
|
export function nativeSelection(input: Record<string, unknown>, game: string, options: Pick<PlacementOptions, 'planned'> = {}): NativeSelection {
|
||||||
|
const selection = nativeLaunchSelection(input, game);
|
||||||
|
const { host } = choosePlacement({ hostId: selection.hostId, libraryId: selection.libraryId,
|
||||||
|
image: selection.image!, bindAddress: selection.bindAddress, ...options });
|
||||||
|
return { ...selection, hostId: host.id, bindAddress: selection.bindAddress ?? host.bindAddress,
|
||||||
|
publicAddress: selection.publicAddress ?? host.publicAddress };
|
||||||
}
|
}
|
||||||
|
|
||||||
export function previewNativeCreation(input: Record<string, unknown>): LaunchPreview {
|
export function previewNativeCreation(input: Record<string, unknown>): LaunchPreview {
|
||||||
@@ -79,7 +93,7 @@ export function previewNativeCreation(input: Record<string, unknown>): LaunchPre
|
|||||||
db().prepare('INSERT INTO server_launch_previews(id,expires_at,server,port,recipe,blueprint_name) VALUES(?,?,?,?,?,?)')
|
db().prepare('INSERT INTO server_launch_previews(id,expires_at,server,port,recipe,blueprint_name) VALUES(?,?,?,?,?,?)')
|
||||||
.run(id, expiresAt, name, port, JSON.stringify(recipe), blueprintName);
|
.run(id, expiresAt, name, port, JSON.stringify(recipe), blueprintName);
|
||||||
const checks: LaunchPreview['checks'] = [
|
const checks: LaunchPreview['checks'] = [
|
||||||
{ label: 'Host capacity', state: 'pass', detail: selection.automatic ? 'Capacity available. The least loaded eligible host is selected when provisioning starts.' : `${getHost(selection.hostId)!.name}: capacity available.` },
|
{ label: 'Host capacity', state: 'pass', detail: selection.automatic ? 'Capacity available. The least loaded eligible host is selected when provisioning starts.' : `${getHost(selection.hostId!)!.name}: capacity available.` },
|
||||||
{ label: 'Game files', state: 'pass', detail: `The latest ${recipe.game.toUpperCase()} game files are verified on an eligible host.` },
|
{ label: 'Game files', state: 'pass', detail: `The latest ${recipe.game.toUpperCase()} game files are verified on an eligible host.` },
|
||||||
{ label: 'Lobby', state: name ? 'pass' : 'blocked', detail: name ?? 'All lobby IDs for this map are reserved.' },
|
{ label: 'Lobby', state: name ? 'pass' : 'blocked', detail: name ?? 'All lobby IDs for this map are reserved.' },
|
||||||
{ label: 'Host port', state: port ? 'pass' : 'blocked', detail: port ? selection.automatic ? 'UDP/TCP capacity available; port assigned at placement.' : `${selection.bindAddress}:${port} / UDP and TCP` : 'No available UDP/TCP port on this host.' },
|
{ label: 'Host port', state: port ? 'pass' : 'blocked', detail: port ? selection.automatic ? 'UDP/TCP capacity available; port assigned at placement.' : `${selection.bindAddress}:${port} / UDP and TCP` : 'No available UDP/TCP port on this host.' },
|
||||||
@@ -90,7 +104,7 @@ export function previewNativeCreation(input: Record<string, unknown>): LaunchPre
|
|||||||
}
|
}
|
||||||
|
|
||||||
export function nextNativePort(selection: NativeSelection, taken: number[] = []): number | null {
|
export function nextNativePort(selection: NativeSelection, taken: number[] = []): number | null {
|
||||||
const host = getHost(selection.hostId);
|
const host = selection.hostId ? getHost(selection.hostId) : null;
|
||||||
return host ? availableHostPort(host, { bindAddress: selection.bindAddress,
|
return host ? availableHostPort(host, { bindAddress: selection.bindAddress,
|
||||||
planned: taken.map(port => ({ hostId: host.id, port })) }) : null;
|
planned: taken.map(port => ({ hostId: host.id, port })) }) : null;
|
||||||
}
|
}
|
||||||
@@ -99,42 +113,51 @@ export function queueNativeCreation(input: CreateServerInput): CreateServerResul
|
|||||||
try {
|
try {
|
||||||
return db().transaction(() => {
|
return db().transaction(() => {
|
||||||
const preview = input.previewId ? db().prepare('SELECT * FROM server_launch_previews WHERE id=?').get(input.previewId) as
|
const preview = input.previewId ? db().prepare('SELECT * FROM server_launch_previews WHERE id=?').get(input.previewId) as
|
||||||
{ id: string; recipe: string; server: string; port: number; expires_at: string; deployment_id: string | null; blueprint_name: string | null } | undefined : undefined;
|
{ id: string; recipe: string; server: string; port: number | null; expires_at: string; deployment_id: string | null; blueprint_name: string | null } | undefined : undefined;
|
||||||
if (input.previewId && !preview) throw new Error('Review a fresh launch preview.');
|
if (input.previewId && !preview) throw new Error('Review a fresh launch preview.');
|
||||||
if (preview?.deployment_id) return { ok: true as const, name: preview.server, port: preview.port, deploymentId: preview.deployment_id };
|
if (preview?.deployment_id) return { ok: true as const, name: preview.server, port: preview.port, deploymentId: preview.deployment_id };
|
||||||
if (preview && preview.expires_at <= new Date().toISOString()) throw new Error('Launch preview expired.');
|
if (preview && preview.expires_at <= new Date().toISOString()) throw new Error('Launch preview expired.');
|
||||||
const saved = preview ? JSON.parse(preview.recipe) as NativeRecipe : undefined;
|
const saved = preview ? JSON.parse(preview.recipe) as NativeRecipe : undefined;
|
||||||
const game = input.game ?? 't5';
|
const game = input.game ?? 't5';
|
||||||
if (input.gameMode !== 'zombie' || (saved && (saved.game !== game || saved.mapSlug !== input.mapSlug))) throw new Error('Selection changed. Review a fresh launch preview.');
|
if (input.gameMode !== 'zombie' || (saved && (saved.game !== game || saved.mapSlug !== input.mapSlug))) throw new Error('Selection changed. Review a fresh launch preview.');
|
||||||
const selection = nativeSelection((saved?.native ?? input) as unknown as Record<string, unknown>, game);
|
const selection = nativeLaunchSelection((saved?.native ?? input) as unknown as Record<string, unknown>, game);
|
||||||
const recipe: NativeRecipe = { ...captureRecipePlan(game, input.mapSlug), native: selection, image: selection.image! };
|
|
||||||
const lobby = nextLobbyNumber(`${game}_zombie_${input.mapSlug}`);
|
const lobby = nextLobbyNumber(`${game}_zombie_${input.mapSlug}`);
|
||||||
if (lobby === null) throw new Error('All lobby IDs are reserved.');
|
if (lobby === null) throw new Error('All lobby IDs are reserved.');
|
||||||
const name = `${game}_zombie_${input.mapSlug}_${lobby}`;
|
const name = `${game}_zombie_${input.mapSlug}_${lobby}`;
|
||||||
const port = nextNativePort(selection);
|
if ((input.expectedName && input.expectedName !== name) || (preview && preview.server !== name)) throw new Error('Available lobby changed. Review a fresh preview.');
|
||||||
if (!port) throw new Error('No available host port.');
|
const result = queueNativeCreationIntent({ server: name, game, mapSlug: input.mapSlug, selection,
|
||||||
if ((input.expectedName && input.expectedName !== name) || (preview && (preview.server !== name || (!selection.automatic && preview.port !== port)))) throw new Error('Available lobby or host port changed. Review a fresh preview.');
|
plutoKey: input.plutoKey, createdBy: input.createdBy, blueprintName: preview?.blueprint_name,
|
||||||
const key = input.plutoKey?.trim() || getServerKey(name);
|
recipe: saved ?? captureRecipePlan(game, input.mapSlug) });
|
||||||
if (!key) throw new Error(`Save a Plutonium server key for ${name}.`);
|
if (result.ok && preview) db().prepare('UPDATE server_launch_previews SET deployment_id=?,port=NULL WHERE id=?').run(result.deploymentId, preview.id);
|
||||||
saveServerKey(name, key);
|
return result;
|
||||||
const instanceId = randomUUID();
|
})();
|
||||||
reserveHostPort(selection.hostId, instanceId, { port, bindAddress: selection.bindAddress, protocol: 'udp' });
|
} catch (error) { return { ok: false, kind: 'bad-request', message: (error as Error).message }; }
|
||||||
reserveHostPort(selection.hostId, instanceId, { port, bindAddress: selection.bindAddress, protocol: 'tcp' });
|
}
|
||||||
db().prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,bind_address,port,public_address,public_port,created_at)
|
|
||||||
VALUES(?,?,?,?,?,?,?,?,?,?)`).run(instanceId, name, selection.hostId, selection.libraryId, selection.image, selection.bindAddress, port,
|
/** Persist the reviewed logical identity; all host/port/resource decisions belong to provisioning. */
|
||||||
selection.publicAddress ?? null, selection.publicPort ?? null, new Date().toISOString());
|
export function queueNativeCreationIntent(input: { server: string; game: 't5' | 't6'; mapSlug: string; selection: NativeSelection;
|
||||||
const id = beginDeployment(name, 'create', input.createdBy);
|
plutoKey?: string; createdBy?: string; blueprintName?: string | null; recipe?: LaunchRecipePlan }): CreateServerResult {
|
||||||
|
try {
|
||||||
|
return db().transaction(() => {
|
||||||
|
const expected = `${input.game}_zombie_${input.mapSlug}_${nextLobbyNumber(`${input.game}_zombie_${input.mapSlug}`)}`;
|
||||||
|
if (input.server !== expected || nativeInstance(input.server)) throw new Error('Available lobby changed. Review a fresh preview.');
|
||||||
|
const key = input.plutoKey?.trim() || getServerKey(input.server);
|
||||||
|
if (!key) throw new Error(`Save a Plutonium server key for ${input.server}.`);
|
||||||
|
saveServerKey(input.server, key);
|
||||||
|
const recipe: NativeRecipe = { ...(input.recipe ?? captureRecipePlan(input.game, input.mapSlug)),
|
||||||
|
native: input.selection, image: input.selection.image! };
|
||||||
|
const id = beginDeployment(input.server, 'create', input.createdBy ?? 'ui');
|
||||||
db().prepare("UPDATE server_deployments SET state='queued' WHERE id=?").run(id);
|
db().prepare("UPDATE server_deployments SET state='queued' WHERE id=?").run(id);
|
||||||
db().prepare('INSERT INTO server_creation_jobs(deployment_id,server,port,game_mode,map_slug,expected_map) VALUES(?,?,?,?,?,?)')
|
db().prepare('INSERT INTO server_creation_jobs(deployment_id,server,port,game_mode,map_slug,expected_map) VALUES(?,?,NULL,?,?,?)')
|
||||||
.run(id, name, port, 'zombie', recipe.mapSlug, recipe.expectedMap);
|
.run(id, input.server, 'zombie', input.mapSlug, recipe.expectedMap);
|
||||||
|
db().prepare('INSERT INTO native_creation_intents(deployment_id,selection) VALUES(?,?)').run(id, JSON.stringify(input.selection));
|
||||||
const stored = recipeStorage(recipe);
|
const stored = recipeStorage(recipe);
|
||||||
db().prepare('INSERT INTO server_launch_recipes(deployment_id,recipe,recipe_summary,blueprint_name) VALUES(?,?,?,?)')
|
db().prepare('INSERT INTO server_launch_recipes(deployment_id,recipe,recipe_summary,blueprint_name) VALUES(?,?,?,?)')
|
||||||
.run(id, stored.recipe, stored.recipeSummary, preview?.blueprint_name ?? null);
|
.run(id, stored.recipe, stored.recipeSummary, input.blueprintName ?? null);
|
||||||
if (preview) db().prepare('UPDATE server_launch_previews SET deployment_id=?,port=? WHERE id=?').run(id, port, preview.id);
|
deploymentStage(id, 'Awaiting placement', { phase: 'eligibility', recipeStatus: 'queued' });
|
||||||
deploymentStage(id, 'Waiting for provisioning evaluation', { phase: 'eligibility', recipeStatus: 'queued' });
|
return { ok: true as const, name: input.server, port: null, deploymentId: id };
|
||||||
return { ok: true as const, name, port, deploymentId: id };
|
|
||||||
})();
|
})();
|
||||||
} catch (error) { return { ok: false, kind: error instanceof CapacityError ? 'conflict' : 'bad-request', message: (error as Error).message }; }
|
} catch (error) { return { ok: false, kind: 'bad-request', message: (error as Error).message }; }
|
||||||
}
|
}
|
||||||
|
|
||||||
export function bindNativeOperation(id: string, server: string): void {
|
export function bindNativeOperation(id: string, server: string): void {
|
||||||
@@ -144,8 +167,11 @@ export function bindNativeOperation(id: string, server: string): void {
|
|||||||
|
|
||||||
/** Cancelling an unsubmitted launch releases only its local reservation. */
|
/** Cancelling an unsubmitted launch releases only its local reservation. */
|
||||||
export function cancelQueuedNativeCreation(server: string, triggeredBy: string): string | null {
|
export function cancelQueuedNativeCreation(server: string, triggeredBy: string): string | null {
|
||||||
|
if (getServer(server)) return null;
|
||||||
|
const unplaced = unplacedNativeCreation(server);
|
||||||
|
if (unplaced) return cancelCreation(unplaced, triggeredBy);
|
||||||
const instance = nativeInstance(server);
|
const instance = nativeInstance(server);
|
||||||
if (!instance || getServer(server)) return null;
|
if (!instance) return null;
|
||||||
return db().transaction(() => {
|
return db().transaction(() => {
|
||||||
const creation = db().prepare(`SELECT d.id FROM server_deployments d JOIN native_provisioning n ON n.deployment_id=d.id
|
const creation = db().prepare(`SELECT d.id FROM server_deployments d JOIN native_provisioning n ON n.deployment_id=d.id
|
||||||
WHERE n.instance_id=? AND d.action='create' AND d.state='queued' AND n.operation_id IS NULL`).get(instance.id) as { id: string } | undefined;
|
WHERE n.instance_id=? AND d.action='create' AND d.state='queued' AND n.operation_id IS NULL`).get(instance.id) as { id: string } | undefined;
|
||||||
@@ -243,9 +269,10 @@ export async function processNativeProvisioning(eligibility: (operation: { id: s
|
|||||||
deploymentStage(id, 'Host draining; waiting for a fresh empty lobby');
|
deploymentStage(id, 'Host draining; waiting for a fresh empty lobby');
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
const jobs = db().prepare(`SELECT d.*,n.instance_id,n.generation,n.stage_operation_id,n.operation_id,n.files_json,n.applied_at
|
const jobs = db().prepare(`SELECT d.*,n.instance_id,n.generation,n.stage_operation_id,n.operation_id,n.files_json,n.applied_at,q.selection
|
||||||
FROM native_provisioning n JOIN server_deployments d ON d.id=n.deployment_id
|
FROM server_deployments d LEFT JOIN native_provisioning n ON n.deployment_id=d.id
|
||||||
WHERE d.server_deleted_at IS NULL AND d.state IN ('queued','waiting','running','awaiting-health','interrupted') ORDER BY d.started_at,d.rowid`).all() as Job[];
|
LEFT JOIN native_creation_intents q ON q.deployment_id=d.id
|
||||||
|
WHERE (n.deployment_id IS NOT NULL OR q.deployment_id IS NOT NULL) AND d.server_deleted_at IS NULL AND d.state IN ('queued','waiting','running','awaiting-health','interrupted') ORDER BY d.started_at,d.rowid`).all() as Job[];
|
||||||
// One host's observation must not prevent another job being admitted.
|
// One host's observation must not prevent another job being admitted.
|
||||||
// Per-job ownership also covers overlapping direct calls and worker passes.
|
// Per-job ownership also covers overlapping direct calls and worker passes.
|
||||||
const work = jobs.map(job => {
|
const work = jobs.map(job => {
|
||||||
@@ -254,7 +281,7 @@ export async function processNativeProvisioning(eligibility: (operation: { id: s
|
|||||||
const work = trackActivity(`provisioning:${job.id}`, `Evaluating ${job.action} for ${job.server}`,
|
const work = trackActivity(`provisioning:${job.id}`, `Evaluating ${job.action} for ${job.server}`,
|
||||||
() => processJob(job, eligibility))
|
() => processJob(job, eligibility))
|
||||||
.catch(error => { if (stillActive(job)) hold(job, (error as Error).message); })
|
.catch(error => { if (stillActive(job)) hold(job, (error as Error).message); })
|
||||||
.finally(() => { activeJobs.delete(job.id); });
|
.finally(() => { activeJobs.delete(job.id); preparingJobs.delete(job.id); });
|
||||||
activeJobs.set(job.id, work);
|
activeJobs.set(job.id, work);
|
||||||
return work;
|
return work;
|
||||||
});
|
});
|
||||||
@@ -281,27 +308,47 @@ function sameLaunchFiles(first: LaunchRecipePlan, second: LaunchRecipePlan): boo
|
|||||||
return JSON.stringify(selected(first)) === JSON.stringify(selected(second));
|
return JSON.stringify(selected(first)) === JSON.stringify(selected(second));
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Reservations can move only while no remote work has ever been submitted. */
|
/** Capacity history and host ranking run in the background reader, one job at a time. */
|
||||||
function placeQueuedCreation(job: Job, instance: NativeInstance): NativeInstance {
|
async function placeQueuedCreation(job: Job, instance: NativeInstance | null): Promise<NativeInstance | null> {
|
||||||
return db().transaction(() => {
|
|
||||||
const recipe = recipeForDeployment(job.id)!.recipe as NativeRecipe;
|
const recipe = recipeForDeployment(job.id)!.recipe as NativeRecipe;
|
||||||
const prior = recipe.native!;
|
const prior: NativeSelection = job.selection ? JSON.parse(job.selection) : recipe.native!;
|
||||||
const network = prior.automatic ? prior.network ?? {} : prior;
|
const network = prior.automatic ? prior.network ?? {} : prior;
|
||||||
const { host, port } = choosePlacement({ hostId: prior.automatic ? undefined : instance.host_id,
|
const options: PlacementOptions = { hostId: prior.automatic ? undefined : prior.hostId,
|
||||||
libraryId: instance.library_id, image: instance.image, bindAddress: network.bindAddress,
|
libraryId: prior.libraryId, image: prior.image ?? recipe.image, bindAddress: network.bindAddress,
|
||||||
thorough: true, dispatch: true, excludeInstanceId: instance.id });
|
thorough: true, dispatch: true, excludeInstanceId: instance?.id };
|
||||||
|
const decision = await evaluateNativePlacement(options);
|
||||||
|
return db().transaction(() => {
|
||||||
|
const state = db().prepare('SELECT state FROM server_deployments WHERE id=? AND server_deleted_at IS NULL').get(job.id) as { state: string } | undefined;
|
||||||
|
if (state?.state !== 'queued') return null;
|
||||||
|
const live = nativeInstance(job.server);
|
||||||
|
if (instance ? live?.id !== instance.id || live.generation !== instance.generation : !!live || !!getServer(job.server)) return null;
|
||||||
|
const active = (db().prepare(`SELECT count(*) AS n FROM server_deployments WHERE id!=? AND server_deleted_at IS NULL
|
||||||
|
AND action IN ('create','rebuild') AND state IN ('running','awaiting-health')`).get(job.id) as { n: number }).n;
|
||||||
|
if (active + otherPreparations(job.id) >= fleetSettings().createConcurrency) { hold(job, 'Waiting for startup capacity.'); return null; }
|
||||||
|
if (getHost(decision.hostId)?.lastHeartbeat !== decision.lastHeartbeat) { hold(job, 'Host measurements changed; waiting for another placement check.'); return null; }
|
||||||
|
// A short current-state check protects against reservations made while the
|
||||||
|
// reader ranked hosts. It never scans resource history or contacts agents.
|
||||||
|
const { host, port } = choosePlacement({ ...options, hostId: decision.hostId, thorough: false });
|
||||||
const bind = network.bindAddress ?? host.bindAddress;
|
const bind = network.bindAddress ?? host.bindAddress;
|
||||||
releaseHostPorts(instance.id);
|
const instanceId = instance?.id ?? randomUUID();
|
||||||
reserveHostPort(host.id, instance.id, { port, bindAddress: bind, protocol: 'udp' });
|
if (instance) releaseHostPorts(instanceId);
|
||||||
reserveHostPort(host.id, instance.id, { port, bindAddress: bind, protocol: 'tcp' });
|
reserveHostPort(host.id, instanceId, { port, bindAddress: bind, protocol: 'udp' });
|
||||||
db().prepare(`UPDATE native_instances SET host_id=?,bind_address=?,port=?,public_address=?,public_port=? WHERE id=?`)
|
reserveHostPort(host.id, instanceId, { port, bindAddress: bind, protocol: 'tcp' });
|
||||||
.run(host.id, bind, port, network.publicAddress ?? host.publicAddress, network.publicPort ?? null, instance.id);
|
if (instance) db().prepare(`UPDATE native_instances SET host_id=?,bind_address=?,port=?,public_address=?,public_port=? WHERE id=?`)
|
||||||
|
.run(host.id, bind, port, network.publicAddress ?? host.publicAddress, network.publicPort ?? null, instanceId);
|
||||||
|
else {
|
||||||
|
db().prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,bind_address,port,public_address,public_port,created_at)
|
||||||
|
VALUES(?,?,?,?,?,?,?,?,?,?)`).run(instanceId, job.server, host.id, prior.libraryId, options.image, bind, port,
|
||||||
|
network.publicAddress ?? host.publicAddress, network.publicPort ?? null, new Date().toISOString());
|
||||||
|
db().prepare('INSERT INTO native_provisioning(deployment_id,instance_id,generation) VALUES(?,?,1)').run(job.id, instanceId);
|
||||||
|
}
|
||||||
db().prepare('UPDATE server_creation_jobs SET port=? WHERE deployment_id=?').run(port, job.id);
|
db().prepare('UPDATE server_creation_jobs SET port=? WHERE deployment_id=?').run(port, job.id);
|
||||||
recipe.native = { ...prior, hostId: host.id, bindAddress: bind, publicAddress: network.publicAddress ?? host.publicAddress, publicPort: network.publicPort };
|
recipe.native = { ...prior, hostId: host.id, bindAddress: bind, publicAddress: network.publicAddress ?? host.publicAddress, publicPort: network.publicPort };
|
||||||
const stored = recipeStorage(recipe);
|
const stored = recipeStorage(recipe);
|
||||||
db().prepare('UPDATE server_launch_recipes SET recipe=?,recipe_summary=? WHERE deployment_id=?').run(stored.recipe, stored.recipeSummary, job.id);
|
db().prepare('UPDATE server_launch_recipes SET recipe=?,recipe_summary=? WHERE deployment_id=?').run(stored.recipe, stored.recipeSummary, job.id);
|
||||||
db().prepare('UPDATE server_launch_previews SET port=? WHERE deployment_id=?').run(port, job.id);
|
db().prepare('UPDATE server_launch_previews SET port=? WHERE deployment_id=?').run(port, job.id);
|
||||||
deploymentStage(job.id, `Placed on ${host.name} after CPU, memory, history and port checks`, { phase: 'eligibility' });
|
deploymentStage(job.id, `Placed on ${host.name} after CPU, memory, history and port checks`, { phase: 'eligibility' });
|
||||||
|
preparingJobs.add(job.id);
|
||||||
return nativeInstance(job.server)!;
|
return nativeInstance(job.server)!;
|
||||||
})();
|
})();
|
||||||
}
|
}
|
||||||
@@ -309,15 +356,24 @@ function placeQueuedCreation(job: Job, instance: NativeInstance): NativeInstance
|
|||||||
async function processJob(job: Job, eligibility: (job: Job) => string | null) {
|
async function processJob(job: Job, eligibility: (job: Job) => string | null) {
|
||||||
if (!stillActive(job)) return;
|
if (!stillActive(job)) return;
|
||||||
let instance = nativeInstance(job.server);
|
let instance = nativeInstance(job.server);
|
||||||
if (!instance || instance.id !== job.instance_id || (instance.generation !== job.generation && !(job.action === 'rebuild' && instance.generation === job.generation + 1))) {
|
if (job.action === 'create' && !job.operation_id && !job.stage_operation_id && !job.submitted_at) {
|
||||||
|
// A resumed bound job owns its original identity. An unplaced intent
|
||||||
|
// cannot adopt an instance or registered server that appeared later.
|
||||||
|
if (job.instance_id ? !instance || instance.id !== job.instance_id || instance.generation !== job.generation
|
||||||
|
: !!instance || !!getServer(job.server)) {
|
||||||
finishDeployment(job.id, { ok: false, message: 'Original instance was retired or replaced; no command was sent.' }); return;
|
finishDeployment(job.id, { ok: false, message: 'Original instance was retired or replaced; no command was sent.' }); return;
|
||||||
}
|
}
|
||||||
if (job.action === 'create' && !job.operation_id && !job.stage_operation_id && !job.submitted_at) {
|
|
||||||
const settings = fleetSettings();
|
const settings = fleetSettings();
|
||||||
const active = db().prepare(`SELECT COUNT(*) AS n FROM server_deployments WHERE id!=? AND server_deleted_at IS NULL
|
const active = db().prepare(`SELECT COUNT(*) AS n FROM server_deployments WHERE id!=? AND server_deleted_at IS NULL
|
||||||
AND action IN ('create','rebuild') AND state IN ('running','awaiting-health')`).get(job.id) as { n: number };
|
AND action IN ('create','rebuild') AND state IN ('running','awaiting-health')`).get(job.id) as { n: number };
|
||||||
if (active.n + otherPreparations(job.id) >= settings.createConcurrency) { hold(job, 'Waiting for startup capacity.'); return; }
|
if (active.n + otherPreparations(job.id) >= settings.createConcurrency) { hold(job, 'Waiting for startup capacity.'); return; }
|
||||||
instance = placeQueuedCreation(job, instance);
|
if (eligibility(job)) return;
|
||||||
|
instance = await placeQueuedCreation(job, instance);
|
||||||
|
if (!instance) return;
|
||||||
|
job.instance_id = instance.id; job.generation = instance.generation;
|
||||||
|
}
|
||||||
|
if (!instance || instance.id !== job.instance_id || (instance.generation !== job.generation && !(job.action === 'rebuild' && instance.generation === job.generation + 1))) {
|
||||||
|
finishDeployment(job.id, { ok: false, message: 'Original instance was retired or replaced; no command was sent.' }); return;
|
||||||
}
|
}
|
||||||
const host = getHost(instance.host_id);
|
const host = getHost(instance.host_id);
|
||||||
if (!host?.connected || host.mode === 'disabled') {
|
if (!host?.connected || host.mode === 'disabled') {
|
||||||
@@ -448,7 +504,7 @@ async function processJob(job: Job, eligibility: (job: Job) => string | null) {
|
|||||||
if (job.action === 'destroy' && count(['destroy']) >= settings.destroyConcurrency) { hold(job, 'Waiting for destruction capacity.'); return; }
|
if (job.action === 'destroy' && count(['destroy']) >= settings.destroyConcurrency) { hold(job, 'Waiting for destruction capacity.'); return; }
|
||||||
deploymentStage(job.id, 'Checking host eligibility', { phase: 'eligibility', blocker: null });
|
deploymentStage(job.id, 'Checking host eligibility', { phase: 'eligibility', blocker: null });
|
||||||
if (job.action === 'create') choosePlacement({ hostId: instance.host_id, libraryId: instance.library_id, image: instance.image,
|
if (job.action === 'create') choosePlacement({ hostId: instance.host_id, libraryId: instance.library_id, image: instance.image,
|
||||||
bindAddress: instance.bind_address, excludeInstanceId: instance.id, thorough: true, dispatch: true });
|
bindAddress: instance.bind_address, excludeInstanceId: instance.id, thorough: false, dispatch: true });
|
||||||
if (job.action === 'create' || job.action === 'rebuild') assertReadyLibrary(instance.host_id, instance.library_id);
|
if (job.action === 'create' || job.action === 'rebuild') assertReadyLibrary(instance.host_id, instance.library_id);
|
||||||
const checkOccupancy = async (): Promise<boolean> => {
|
const checkOccupancy = async (): Promise<boolean> => {
|
||||||
// Fast controller check keeps occupied lobbies out of the operation slots.
|
// Fast controller check keeps occupied lobbies out of the operation slots.
|
||||||
@@ -514,7 +570,7 @@ async function processJob(job: Job, eligibility: (job: Job) => string | null) {
|
|||||||
}
|
}
|
||||||
assertReadyLibrary(instance!.host_id, instance!.library_id);
|
assertReadyLibrary(instance!.host_id, instance!.library_id);
|
||||||
if (job.action === 'create') choosePlacement({ hostId: instance!.host_id, libraryId: instance!.library_id, image: instance!.image,
|
if (job.action === 'create') choosePlacement({ hostId: instance!.host_id, libraryId: instance!.library_id, image: instance!.image,
|
||||||
bindAddress: instance!.bind_address, excludeInstanceId: instance!.id, thorough: true, dispatch: true });
|
bindAddress: instance!.bind_address, excludeInstanceId: instance!.id, thorough: false, dispatch: true });
|
||||||
}
|
}
|
||||||
if (job.action === 'start' || job.action === 'restart' || job.action === 'rebuild') {
|
if (job.action === 'start' || job.action === 'restart' || job.action === 'rebuild') {
|
||||||
db().prepare(`INSERT INTO restart_rollout(server,requested_at,operation_id) VALUES(?,?,?) ON CONFLICT(server)
|
db().prepare(`INSERT INTO restart_rollout(server,requested_at,operation_id) VALUES(?,?,?) ON CONFLICT(server)
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ import { queueRebuildServer } from './instances.js';
|
|||||||
import { restartRequestInFlight } from './restarts.js';
|
import { restartRequestInFlight } from './restarts.js';
|
||||||
import { runtimeActions, type RuntimeAction } from '../../provisioning.js';
|
import { runtimeActions, type RuntimeAction } from '../../provisioning.js';
|
||||||
import { recordProgress } from './progress.js';
|
import { recordProgress } from './progress.js';
|
||||||
import { nativeInstance } from '../backends/native.js';
|
import { nativeInstance, unplacedNativeCreation } from '../backends/native.js';
|
||||||
|
|
||||||
export interface LifecycleOperation {
|
export interface LifecycleOperation {
|
||||||
id: string; server: string; action: string; state: string; stage: string;
|
id: string; server: string; action: string; state: string; stage: string;
|
||||||
@@ -137,7 +137,7 @@ export function releaseRestartRequest(server: string, source: RestartSource): vo
|
|||||||
|
|
||||||
export function requestLifecycle(server: string, action: RuntimeAction) {
|
export function requestLifecycle(server: string, action: RuntimeAction) {
|
||||||
if (!runtimeActions.includes(action)) return { ok: false as const, status: 400, error: 'Choose start, restart, rebuild, or stop.' };
|
if (!runtimeActions.includes(action)) return { ok: false as const, status: 400, error: 'Choose start, restart, rebuild, or stop.' };
|
||||||
if (!nativeInstance(server) && (!komodoConfigured() || !rconConfigured())) return { ok: false as const, status: 503, error: 'Configure a host and RCON before provisioning.' };
|
if (!nativeInstance(server) && !unplacedNativeCreation(server) && (!komodoConfigured() || !rconConfigured())) return { ok: false as const, status: 503, error: 'Configure a host and RCON before provisioning.' };
|
||||||
return db().transaction(() => {
|
return db().transaction(() => {
|
||||||
const blocker = action === 'rebuild' ? activeBetaRemovalBlocker(server) : null;
|
const blocker = action === 'rebuild' ? activeBetaRemovalBlocker(server) : null;
|
||||||
if (blocker) return { ok: false as const, status: 409, error: blocker };
|
if (blocker) return { ok: false as const, status: 409, error: blocker };
|
||||||
|
|||||||
@@ -31,7 +31,7 @@
|
|||||||
const keysReady = $derived(preview?.servers.filter(server => server.name && (server.hasSavedKey || keys[server.name]?.trim())).length ?? 0);
|
const keysReady = $derived(preview?.servers.filter(server => server.name && (server.hasSavedKey || keys[server.name]?.trim())).length ?? 0);
|
||||||
onMount(() => { const timer = setInterval(() => { now = Date.now(); }, 10_000); return () => clearInterval(timer); });
|
onMount(() => { const timer = setInterval(() => { now = Date.now(); }, 10_000); return () => clearInterval(timer); });
|
||||||
let canLaunch = $derived(preview?.ready && preview.mode === mode && (preview.servers.length > 0 || preview.removals.length > 0)
|
let canLaunch = $derived(preview?.ready && preview.mode === mode && (preview.servers.length > 0 || preview.removals.length > 0)
|
||||||
&& (!preview.servers.length || hostReady) && (!preview.removals.length || confirmRemovals) && preview.servers.every(server => server.name && (server.hasSavedKey || keys[server.name]?.trim())));
|
&& (!preview.removals.length || confirmRemovals) && preview.servers.every(server => server.name && (server.hasSavedKey || keys[server.name]?.trim())));
|
||||||
|
|
||||||
$effect(() => {
|
$effect(() => {
|
||||||
if (!dialog) return;
|
if (!dialog) return;
|
||||||
@@ -54,7 +54,7 @@
|
|||||||
}
|
}
|
||||||
|
|
||||||
async function review() {
|
async function review() {
|
||||||
if (!hostReady) { error = 'Wait for an eligible host and wait for its latest game files and runtime to finish automatic setup.'; return; }
|
if (!hostReady) { error = 'Choose the latest uploaded game files and a valid runtime image before reviewing the launch.'; return; }
|
||||||
const current = ++session;
|
const current = ++session;
|
||||||
const previous = preview;
|
const previous = preview;
|
||||||
busy = true; error = ''; keyNotice = ''; confirmRemovals = false;
|
busy = true; error = ''; keyNotice = ''; confirmRemovals = false;
|
||||||
@@ -105,7 +105,7 @@
|
|||||||
{#if open && games.length}<NativeHostSelection {games} bind:selection={hostSelection} bind:ready={hostReady} disabled={busy} onchange={() => { preview = null; confirmRemovals = false; }} />{/if}
|
{#if open && games.length}<NativeHostSelection {games} bind:selection={hostSelection} bind:ready={hostReady} disabled={busy} onchange={() => { preview = null; confirmRemovals = false; }} />{/if}
|
||||||
<p class="muted">{mode === 'exact' ? 'Keep matching lobbies, add missing ones, and remove excess lobbies and maps outside this collection across both games.' : 'Create missing lobbies and keep all existing servers.'}</p>
|
<p class="muted">{mode === 'exact' ? 'Keep matching lobbies, add missing ones, and remove excess lobbies and maps outside this collection across both games.' : 'Create missing lobbies and keep all existing servers.'}</p>
|
||||||
{#if error}<p class="error" role="alert">{error}</p>{/if}
|
{#if error}<p class="error" role="alert">{error}</p>{/if}
|
||||||
{#if !preview}<p class="muted">{busy ? 'Preparing the collection…' : 'Wait for an eligible host. Its latest game files sync automatically before you review capacity and server keys.'}</p>{#if !busy}<Button disabled={!hostReady} onclick={review}>Preview launch</Button>{/if}
|
{#if !preview}<p class="muted">{busy ? 'Preparing the collection…' : 'Review the fleet changes, capacity estimate, and server keys. Queued launches wait automatically when hosts need more capacity or preparation.'}</p>{#if !busy}<Button disabled={!hostReady} onclick={review}>Preview launch</Button>{/if}
|
||||||
{:else}
|
{:else}
|
||||||
{#if keyNotice}<p role="status" class="muted">{keyNotice}</p>{/if}
|
{#if keyNotice}<p role="status" class="muted">{keyNotice}</p>{/if}
|
||||||
{#if expired}<p class="error" role="status">This preview expired. Refresh before launch; keys remain entered for unchanged IDs.</p>{/if}
|
{#if expired}<p class="error" role="status">This preview expired. Refresh before launch; keys remain entered for unchanged IDs.</p>{/if}
|
||||||
@@ -115,10 +115,24 @@
|
|||||||
{#if preview.mode === 'exact' && preview.retainedServers.length}<details class="coverage"><summary>Servers to keep ({preview.retainedServers.length})</summary><ul>{#each preview.retainedServers as name}<li><ServerIdentity {name} /><span>{name}</span></li>{/each}</ul></details>{/if}
|
{#if preview.mode === 'exact' && preview.retainedServers.length}<details class="coverage"><summary>Servers to keep ({preview.retainedServers.length})</summary><ul>{#each preview.retainedServers as name}<li><ServerIdentity {name} /><span>{name}</span></li>{/each}</ul></details>{/if}
|
||||||
{#if preview.mode === 'fill' && preview.entries.some(entry => entry.existing)}<details class="coverage" open={!preview.servers.length}><summary>Review existing servers</summary>{#each preview.entries.filter(entry => entry.existing) as entry}<div class="coverage-map"><strong>{entry.recipe.mapName}: {entry.existing} existing</strong><p class="muted">{[...new Set((entry.coverage ?? []).map(server => server.health))].map(health => `${entry.coverage.filter(server => server.health === health).length} ${health}`).join(', ')}</p><ul>{#each entry.coverage ?? [] as server}<li><a href={server.href}><ServerIdentity name={server.name} /></a><span>{server.health === 'online' ? 'Healthy' : server.health}</span></li>{/each}</ul></div>{/each}</details>{/if}
|
{#if preview.mode === 'fill' && preview.entries.some(entry => entry.existing)}<details class="coverage" open={!preview.servers.length}><summary>Review existing servers</summary>{#each preview.entries.filter(entry => entry.existing) as entry}<div class="coverage-map"><strong>{entry.recipe.mapName}: {entry.existing} existing</strong><p class="muted">{[...new Set((entry.coverage ?? []).map(server => server.health))].map(health => `${entry.coverage.filter(server => server.health === health).length} ${health}`).join(', ')}</p><ul>{#each entry.coverage ?? [] as server}<li><a href={server.href}><ServerIdentity name={server.name} /></a><span>{server.health === 'online' ? 'Healthy' : server.health}</span></li>{/each}</ul></div>{/each}</details>{/if}
|
||||||
</div>
|
</div>
|
||||||
|
{#if preview.availability && preview.servers.length}
|
||||||
|
<section class="availability" aria-label="Latest reported capacity">
|
||||||
|
<h3>Latest reported capacity</h3>
|
||||||
|
<p class="muted">{preview.availability.estimatedAvailable === null ? 'An estimate is not available yet.' : `${preview.availability.estimatedAvailable} additional servers estimated to fit.`} {preview.availability.requested} requested.</p>
|
||||||
|
<dl>
|
||||||
|
<div><dt>CPU available</dt><dd>{preview.availability.cpuCoresAvailable === null ? 'Unknown' : `${preview.availability.cpuCoresAvailable.toFixed(1)} cores`}</dd></div>
|
||||||
|
<div><dt>Memory available</dt><dd>{preview.availability.memoryBytesAvailable === null ? 'Unknown' : `${(preview.availability.memoryBytesAvailable / 1024 ** 3).toFixed(1)} GiB`}</dd></div>
|
||||||
|
<div><dt>Ports available</dt><dd>{preview.availability.portsAvailable ?? 'Unknown'}</dd></div>
|
||||||
|
<div><dt>Startup slots</dt><dd>{preview.availability.startupSlots ?? 'Unknown'}{#if preview.availability.startupLimit !== null} / {preview.availability.startupLimit}{/if}</dd></div>
|
||||||
|
</dl>
|
||||||
|
<p class="caption">{preview.availability.updatedAt ? `Updated ${$timeFormat.dateTime(preview.availability.updatedAt)}.` : 'Waiting for host reports.'}{#if preview.availability.queued !== null} {preview.availability.queued} servers already queued.{/if} Provisioning checks capacity separately for each server before it starts.</p>
|
||||||
|
</section>
|
||||||
|
{/if}
|
||||||
<div class="section-title"><h3>{preview.mode === 'exact' ? 'Review fleet changes' : preview.servers.length ? 'Your launch order' : 'Nothing to create'}</h3><button class="text-button" disabled={busy} onclick={review}>Refresh preview</button></div>
|
<div class="section-title"><h3>{preview.mode === 'exact' ? 'Review fleet changes' : preview.servers.length ? 'Your launch order' : 'Nothing to create'}</h3><button class="text-button" disabled={busy} onclick={review}>Refresh preview</button></div>
|
||||||
{#each preview.blockers as blocker}<p class="error" role="alert">{blocker}</p>{/each}
|
{#each preview.blockers as blocker}<p class="error" role="alert">{blocker}</p>{/each}
|
||||||
|
{#each preview.warnings ?? [] as warning}<p class="warning" role="status">{warning}</p>{/each}
|
||||||
{#if preview.servers.length || preview.removals.length}
|
{#if preview.servers.length || preview.removals.length}
|
||||||
{#if preview.servers.length}<p class="muted">Saved keys are reused automatically. Add keys only where required below. Update saved keys in <a href="/server-keys" target="_blank" rel="noopener">Keys (opens in a new tab)</a>. Fleet counts and available slots are checked again when you queue the launch.</p>{/if}
|
{#if preview.servers.length}<p class="muted">Saved keys are reused automatically. Add keys only where required below. Update saved keys in <a href="/server-keys" target="_blank" rel="noopener">Keys (opens in a new tab)</a>. Apply saves the reviewed jobs and opens their progress page. Each launch waits for a startup slot and host capacity.</p>{/if}
|
||||||
<form onsubmit={event => { event.preventDefault(); void launch(); }}>
|
<form onsubmit={event => { event.preventDefault(); void launch(); }}>
|
||||||
{#if preview.removals.length}
|
{#if preview.removals.length}
|
||||||
<section class="removals" aria-label="Servers to remove">
|
<section class="removals" aria-label="Servers to remove">
|
||||||
@@ -133,7 +147,7 @@
|
|||||||
{#each preview.servers as server, index (server.name ?? `unavailable-${index}`)}
|
{#each preview.servers as server, index (server.name ?? `unavailable-${index}`)}
|
||||||
<li class="server-row">
|
<li class="server-row">
|
||||||
<span class="order" aria-hidden="true">{String(index + 1).padStart(2, '0')}</span>
|
<span class="order" aria-hidden="true">{String(index + 1).padStart(2, '0')}</span>
|
||||||
<div class="server-identity">{#if index === 0 || preview.servers[index - 1].entryIndex !== server.entryIndex}<span class="map-group">{preview.entries[server.entryIndex].recipe.game.toUpperCase()} · {preview.entries[server.entryIndex].recipe.mapName}</span>{/if}<strong>{#if server.name}<ServerIdentity name={server.name} />{:else}{preview.entries[server.entryIndex].recipe.mapName}{/if}</strong><span class="server-id">{server.name ?? 'No available lobby'} · {server.port === null ? 'No available capacity' : server.native?.automatic ? 'Host and port assigned automatically' : `Port ${server.port}`}</span></div>
|
<div class="server-identity">{#if index === 0 || preview.servers[index - 1].entryIndex !== server.entryIndex}<span class="map-group">{preview.entries[server.entryIndex].recipe.game.toUpperCase()} · {preview.entries[server.entryIndex].recipe.mapName}</span>{/if}<strong>{#if server.name}<ServerIdentity name={server.name} />{:else}{preview.entries[server.entryIndex].recipe.mapName}{/if}</strong><span class="server-id">{server.name ?? 'No available lobby'} · {server.native ? 'Host and port assigned when provisioning starts' : server.port === null ? 'No available port' : `Port ${server.port}`}</span></div>
|
||||||
{#if server.name}
|
{#if server.name}
|
||||||
{#if server.hasSavedKey}<p class="key-ready"><span aria-hidden="true">✓</span> Saved key ready</p>
|
{#if server.hasSavedKey}<p class="key-ready"><span aria-hidden="true">✓</span> Saved key ready</p>
|
||||||
{:else}<label><span>Server key required</span><input aria-label={`Key for ${server.name}`} data-missing={!keys[server.name]?.trim()} type="password" bind:value={keys[server.name]} disabled={busy} maxlength="512" autocomplete="off" placeholder="Paste this server’s key" /></label>{/if}
|
{:else}<label><span>Server key required</span><input aria-label={`Key for ${server.name}`} data-missing={!keys[server.name]?.trim()} type="password" bind:value={keys[server.name]} disabled={busy} maxlength="512" autocomplete="off" placeholder="Paste this server’s key" /></label>{/if}
|
||||||
@@ -151,6 +165,10 @@
|
|||||||
</dialog>
|
</dialog>
|
||||||
|
|
||||||
<style>
|
<style>
|
||||||
|
.availability { display: grid; gap: 12px; padding: 16px; border: 1px solid var(--border-subtle); border-radius: var(--radius-md); }
|
||||||
|
.availability dl { display: grid; grid-template-columns: repeat(auto-fit, minmax(130px, 1fr)); gap: 16px; margin: 0; }
|
||||||
|
.availability dt { font-size: var(--text-xs); color: var(--text-secondary); }.availability dd { margin: 6px 0 0; font-size: 14px; font-variant-numeric: tabular-nums; }
|
||||||
|
.warning { color: var(--state-warning); font-size: 13px; line-height: 1.6; }
|
||||||
select { padding: 10px; border: 1px solid var(--border-control); border-radius: var(--radius-md); background: var(--surface-sunken); color: var(--text-primary); font: inherit; }
|
select { padding: 10px; border: 1px solid var(--border-control); border-radius: var(--radius-md); background: var(--surface-sunken); color: var(--text-primary); font: inherit; }
|
||||||
.removals { display: grid; gap: 12px; padding: 16px; border: 1px solid var(--state-danger); border-radius: var(--radius-md); }
|
.removals { display: grid; gap: 12px; padding: 16px; border: 1px solid var(--state-danger); border-radius: var(--radius-md); }
|
||||||
.removals ul { list-style: none; margin: 0; padding: 0; display: grid; gap: 16px; }
|
.removals ul { list-style: none; margin: 0; padding: 0; display: grid; gap: 16px; }
|
||||||
|
|||||||
@@ -15,7 +15,8 @@
|
|||||||
const latest = $derived(latestGameFiles(libraries));
|
const latest = $derived(latestGameFiles(libraries));
|
||||||
|
|
||||||
function cacheReady(game: ZombieGame) { return !!latest[game]?.caches.some(cache => eligible.some(host => host.id === cache.hostId) && ['ready', 'outdated'].includes(cache.status)); }
|
function cacheReady(game: ZombieGame) { return !!latest[game]?.caches.some(cache => eligible.some(host => host.id === cache.hostId) && ['ready', 'outdated'].includes(cache.status)); }
|
||||||
$effect(() => { ready = !loading && !error && games.length > 0 && games.every(game => selection.libraryIds[game] === latest[game]?.id && cacheReady(game)); });
|
$effect(() => { ready = !loading && !error && /^[a-zA-Z0-9][a-zA-Z0-9._/:@-]{0,499}$/.test(selection.image)
|
||||||
|
&& games.length > 0 && games.every(game => !!latest[game]?.id && selection.libraryIds[game] === latest[game]?.id); });
|
||||||
$effect(() => {
|
$effect(() => {
|
||||||
if (disabled || !defaultRuntimeImage || runtimeSelection === selection) return;
|
if (disabled || !defaultRuntimeImage || runtimeSelection === selection) return;
|
||||||
if (!selection.image) selection.image = defaultRuntimeImage;
|
if (!selection.image) selection.image = defaultRuntimeImage;
|
||||||
@@ -42,14 +43,14 @@
|
|||||||
<div class="title"><h3>Host and game files</h3><button type="button" disabled={disabled || loading} onclick={() => { onchange?.(); void load(); }}>Refresh hosts</button></div>
|
<div class="title"><h3>Host and game files</h3><button type="button" disabled={disabled || loading} onclick={() => { onchange?.(); void load(); }}>Refresh hosts</button></div>
|
||||||
{#if error}<p role="alert" class="error">{error}</p>{/if}
|
{#if error}<p role="alert" class="error">{error}</p>{/if}
|
||||||
{#if loading}<p role="status">Loading host readiness and game files…</p>{/if}
|
{#if loading}<p role="status">Loading host readiness and game files…</p>{/if}
|
||||||
<p>Hosts are assigned automatically. We check available capacity before accepting a launch and choose the least loaded eligible host when provisioning starts.</p>
|
<p>Hosts are assigned when provisioning starts. You can queue launches while hosts are busy or preparing; each server waits for an eligible host and available capacity.</p>
|
||||||
{#if !loading && !hosts.length}<p><a href="/hosts" target="_blank" rel="noopener">Add a host</a>. Its runtime and latest T5 and T6 files will be prepared automatically.</p>{/if}
|
{#if !loading && !hosts.length}<p><a href="/hosts" target="_blank" rel="noopener">Add a host</a>. Its runtime and latest T5 and T6 files will be prepared automatically.</p>{/if}
|
||||||
{#if !loading}{#each games as game}
|
{#if !loading}{#each games as game}
|
||||||
<div class="game-files">
|
<div class="game-files">
|
||||||
<strong>{game === 't5' ? 'Black Ops I (T5)' : 'Black Ops II (T6)'} files</strong>
|
<strong>{game === 't5' ? 'Black Ops I (T5)' : 'Black Ops II (T6)'} files</strong>
|
||||||
{#if latest[game]}
|
{#if latest[game]}
|
||||||
<p>Latest upload · {$timeFormat.dateTime(latest[game]?.publishedAt ?? latest[game]?.createdAt)}</p>
|
<p>Latest upload · {$timeFormat.dateTime(latest[game]?.publishedAt ?? latest[game]?.createdAt)}</p>
|
||||||
<p role="status">{cacheReady(game) ? 'Latest files ready on an eligible host.' : 'Waiting for a host to prepare the runtime and latest files automatically.'}</p>
|
<p role="status">{cacheReady(game) ? 'Latest files ready on an eligible host.' : 'Launches can be queued now and will wait for a host to prepare the runtime and latest files.'}</p>
|
||||||
{:else}
|
{:else}
|
||||||
<p>No files uploaded yet. <a href="/game-files" target="_blank" rel="noopener">Upload {game.toUpperCase()} files</a> to prepare your hosts.</p>
|
<p>No files uploaded yet. <a href="/game-files" target="_blank" rel="noopener">Upload {game.toUpperCase()} files</a> to prepare your hosts.</p>
|
||||||
{/if}
|
{/if}
|
||||||
|
|||||||
@@ -24,17 +24,15 @@
|
|||||||
const waitingForPort = (server: CollectionServer) => server.state === 'queued' && (server.stage.startsWith('Waiting for removal') || server.stage.startsWith('Waiting for port'));
|
const waitingForPort = (server: CollectionServer) => server.state === 'queued' && (server.stage.startsWith('Waiting for removal') || server.stage.startsWith('Waiting for port'));
|
||||||
function stateLabel(server: CollectionServer) {
|
function stateLabel(server: CollectionServer) {
|
||||||
if (waitingForPort(server)) return 'Waiting for removal';
|
if (waitingForPort(server)) return 'Waiting for removal';
|
||||||
if (server.state === 'queued' && server.queuePosition === 1) return 'Up next';
|
|
||||||
if (server.action === 'destroy' && server.state === 'done') return 'Removed';
|
if (server.action === 'destroy' && server.state === 'done') return 'Removed';
|
||||||
if (server.action === 'destroy' && server.state === 'running') return 'Removing';
|
if (server.action === 'destroy' && server.state === 'running') return 'Removing';
|
||||||
return labels[server.state] ?? server.state;
|
return labels[server.state] ?? server.state;
|
||||||
}
|
}
|
||||||
|
|
||||||
function queueHint(server: CollectionServer) {
|
function queueHint(server: CollectionServer) {
|
||||||
if (waitingForPort(server)) return server.stage;
|
const reason = server.stage || 'Waiting for a startup slot and available host capacity.';
|
||||||
if (server.queuePosition === 1) return 'Next in the launch queue. Starts when a place opens.';
|
return server.queuePosition && server.queuePosition > 1
|
||||||
if (server.queuePosition) return `${server.queuePosition - 1} queued server${server.queuePosition === 2 ? '' : 's'} ahead. Starts automatically when it is this lobby’s turn.`;
|
? `${reason} ${server.queuePosition - 1} queued server${server.queuePosition === 2 ? '' : 's'} ahead.` : reason;
|
||||||
return 'Waiting for its turn in the launch queue.';
|
|
||||||
}
|
}
|
||||||
|
|
||||||
async function load() {
|
async function load() {
|
||||||
@@ -90,11 +88,12 @@
|
|||||||
{:else if firstQueued}
|
{:else if firstQueued}
|
||||||
<aside class="waiting-note">
|
<aside class="waiting-note">
|
||||||
<span class="waiting-icon" aria-hidden="true">◷</span>
|
<span class="waiting-icon" aria-hidden="true">◷</span>
|
||||||
<div><strong>{waitingForPort(firstQueued) ? 'Waiting for a removal to release capacity' : otherLaunches.length ? 'Waiting for an earlier launch' : firstQueued?.queuePosition && firstQueued.queuePosition > 1 ? 'Earlier servers are ahead in the queue' : 'Your launch is up next'}</strong>
|
<div><strong>{waitingForPort(firstQueued) ? 'Waiting for a removal to release capacity' : 'Your collection is queued'}</strong>
|
||||||
|
<p>{queueHint(firstQueued)} Provisioning starts each server when its host and capacity are ready. Progress updates automatically.</p>
|
||||||
{#if otherLaunches.length}
|
{#if otherLaunches.length}
|
||||||
<p>Your collection will start automatically as earlier launches finish. Currently launching:</p>
|
<p>Other servers currently launching:</p>
|
||||||
<div class="other-launches">{#each otherLaunches as active (active.id)}<a href="/provisioning/{active.id}"><ServerIdentity name={active.server} /> <span aria-hidden="true">→</span></a>{/each}</div>
|
<div class="other-launches">{#each otherLaunches as active (active.id)}<a href="/provisioning/{active.id}"><ServerIdentity name={active.server} /> <span aria-hidden="true">→</span></a>{/each}</div>
|
||||||
{:else}<p>{firstQueued ? queueHint(firstQueued) : 'Your servers will begin automatically.'} Progress updates automatically.</p>{/if}
|
{/if}
|
||||||
</div>
|
</div>
|
||||||
</aside>
|
</aside>
|
||||||
{/if}
|
{/if}
|
||||||
@@ -114,7 +113,7 @@
|
|||||||
<div class="server-content">
|
<div class="server-content">
|
||||||
<div class="server-heading"><strong><HistoricalServerIdentity name={server.server} deletedAt={server.server_deleted_at} /></strong><Badge variant={needsAttention(server) ? 'danger' : server.state === 'done' ? 'online' : launching(server) ? 'info' : 'neutral'}>{stateLabel(server)}</Badge></div>
|
<div class="server-heading"><strong><HistoricalServerIdentity name={server.server} deletedAt={server.server_deleted_at} /></strong><Badge variant={needsAttention(server) ? 'danger' : server.state === 'done' ? 'online' : launching(server) ? 'info' : 'neutral'}>{stateLabel(server)}</Badge></div>
|
||||||
<p class:error={!!server.error}>{server.error ?? (server.state === 'queued' ? queueHint(server) : server.state === 'done' ? server.action === 'destroy' ? server.stage : 'Startup checks passed.' : server.stage)}</p>
|
<p class:error={!!server.error}>{server.error ?? (server.state === 'queued' ? queueHint(server) : server.state === 'done' ? server.action === 'destroy' ? server.stage : 'Startup checks passed.' : server.stage)}</p>
|
||||||
<div class="server-meta"><span>{server.action === 'destroy' ? 'Remove server' : `Create server · Port ${server.port}`}</span><span>{needsAttention(server) ? 'Review & recover' : 'View operation'} <span aria-hidden="true">→</span></span></div>
|
<div class="server-meta"><span>{server.action === 'destroy' ? 'Remove server' : server.port === null ? 'Create server · Port not assigned yet' : `Create server · Port ${server.port}`}</span><span>{needsAttention(server) ? 'Review & recover' : 'View operation'} <span aria-hidden="true">→</span></span></div>
|
||||||
</div>
|
</div>
|
||||||
</a>
|
</a>
|
||||||
</li>
|
</li>
|
||||||
|
|||||||
@@ -79,23 +79,46 @@ describe('large native collection admission', () => {
|
|||||||
expect(preview.ready).toBe(true);
|
expect(preview.ready).toBe(true);
|
||||||
expect(preview.servers).toHaveLength(serverCount);
|
expect(preview.servers).toHaveLength(serverCount);
|
||||||
expect(new Set(preview.servers.map(server => server.name)).size).toBe(serverCount);
|
expect(new Set(preview.servers.map(server => server.name)).size).toBe(serverCount);
|
||||||
expect(new Set(preview.servers.map(server => server.port)).size).toBe(serverCount);
|
expect(preview.servers.every(server => server.port === null)).toBe(true);
|
||||||
expect(db().prepare('SELECT * FROM ec_host_ports').all()).toHaveLength(0);
|
expect(db().prepare('SELECT * FROM ec_host_ports').all()).toHaveLength(0);
|
||||||
|
|
||||||
|
const native = await import('../../src/lib/server/provisioning/native.js');
|
||||||
|
const validateSelection = vi.spyOn(native, 'nativeLaunchSelection');
|
||||||
manifestParses = 0;
|
manifestParses = 0;
|
||||||
const result = queueCollection({ previewId: preview.id,
|
const result = queueCollection({ previewId: preview.id,
|
||||||
keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name])) });
|
keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name])) });
|
||||||
const queueParses = manifestParses;
|
const queueParses = manifestParses;
|
||||||
expect(result).toMatchObject({ ok: true, collectionId: expect.any(String) });
|
expect(result).toMatchObject({ ok: true, collectionId: expect.any(String) });
|
||||||
expect(queueCollection({ previewId: preview.id })).toEqual(result);
|
expect(queueCollection({ previewId: preview.id })).toEqual(result);
|
||||||
|
expect(validateSelection).toHaveBeenCalledTimes(1);
|
||||||
expect(getCollection(result.collectionId!)?.collection).toMatchObject({ total: serverCount, queued: serverCount });
|
expect(getCollection(result.collectionId!)?.collection).toMatchObject({ total: serverCount, queued: serverCount });
|
||||||
const instances = db().prepare('SELECT host_id,library_id,port FROM native_instances ORDER BY port').all();
|
expect(db().prepare('SELECT host_id,library_id,port FROM native_instances').all()).toEqual([]);
|
||||||
expect(instances).toEqual(Array.from({ length: serverCount }, (_, index) => ({
|
expect(db().prepare('SELECT * FROM ec_host_ports').all()).toEqual([]);
|
||||||
host_id: hostId, library_id: libraryId, port: 28960 + index
|
const intents = db().prepare('SELECT selection FROM native_creation_intents').all() as { selection: string }[];
|
||||||
})));
|
expect(intents).toHaveLength(serverCount);
|
||||||
expect(db().prepare('SELECT * FROM ec_host_ports').all()).toHaveLength(serverCount * 2);
|
expect(intents.every(intent => JSON.parse(intent.selection).libraryId === libraryId)).toBe(true);
|
||||||
expect(db().prepare('SELECT * FROM server_collection_members').all()).toHaveLength(serverCount);
|
expect(db().prepare('SELECT * FROM server_collection_members').all()).toHaveLength(serverCount);
|
||||||
expect(previewParses, 'Preview must use library metadata without reading the full manifest').toBe(0);
|
expect(previewParses, 'Preview must use library metadata without reading the full manifest').toBe(0);
|
||||||
expect(queueParses, 'Queue admission must use library metadata without reading the full manifest').toBe(0);
|
expect(queueParses, 'Queue admission must use library metadata without reading the full manifest').toBe(0);
|
||||||
}, 30_000);
|
}, 30_000);
|
||||||
|
|
||||||
|
it.each(['newer game files', 'deleted pinned host'])('rejects %s after review before queueing any collection work', async change => {
|
||||||
|
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
|
||||||
|
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const blueprint = saveBlueprint({ name: 'Reviewed native selection', entries: [{ game: 't5', mapSlug: 'theater', count: 2 }] });
|
||||||
|
const preview = previewCollection(blueprint.id, 'fill', { hostId, libraryId, image });
|
||||||
|
if (change === 'newer game files') {
|
||||||
|
const { createLibraryImport } = await import('../../src/lib/server/game-files/index.js');
|
||||||
|
const newer = createLibraryImport({ hostId, game: 't5' });
|
||||||
|
db().prepare("UPDATE game_file_versions SET state='ready',published_at=? WHERE id=?")
|
||||||
|
.run(new Date(Date.now() + 1000).toISOString(), newer.id);
|
||||||
|
} else db().prepare('UPDATE ec_hosts SET deleted_at=? WHERE id=?').run(new Date().toISOString(), hostId);
|
||||||
|
expect(() => queueCollection({ previewId: preview.id,
|
||||||
|
keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name])) }))
|
||||||
|
.toThrow(change === 'newer game files' ? /game files changed/ : /host was not found/);
|
||||||
|
expect(db().prepare('SELECT id FROM server_launch_collections').all()).toEqual([]);
|
||||||
|
expect(db().prepare('SELECT deployment_id FROM native_creation_intents').all()).toEqual([]);
|
||||||
|
expect(db().prepare('SELECT server_id FROM server_keys').all()).toEqual([]);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -12,10 +12,14 @@ vi.mock('../../src/lib/server/integrations/rcon.js', () => ({
|
|||||||
|
|
||||||
let fixture: Fixture;
|
let fixture: Fixture;
|
||||||
let hostId: string;
|
let hostId: string;
|
||||||
|
let agentCredential: string;
|
||||||
|
let heartbeatInventory: Record<string, unknown>;
|
||||||
|
let releasePlacement: (() => void) | undefined;
|
||||||
const image = 'example.test/plutainer:test';
|
const image = 'example.test/plutainer:test';
|
||||||
|
|
||||||
beforeEach(async () => {
|
beforeEach(async () => {
|
||||||
vi.resetModules(); vi.clearAllMocks();
|
vi.resetModules(); vi.clearAllMocks();
|
||||||
|
releasePlacement = undefined;
|
||||||
fixture = await newFixture([]);
|
fixture = await newFixture([]);
|
||||||
vi.stubEnv('GAME_FILES_ROOT', join(fixture.root, 'libraries'));
|
vi.stubEnv('GAME_FILES_ROOT', join(fixture.root, 'libraries'));
|
||||||
vi.stubEnv('PLUTAINER_IMAGE', image);
|
vi.stubEnv('PLUTAINER_IMAGE', image);
|
||||||
@@ -31,7 +35,7 @@ beforeEach(async () => {
|
|||||||
protocolVersion: 1, dockerRunning: true, directoriesWritable: true,
|
protocolVersion: 1, dockerRunning: true, directoriesWritable: true,
|
||||||
storageAvailable: 1_000_000_000_000, runtimeImages: [image]
|
storageAvailable: 1_000_000_000_000, runtimeImages: [image]
|
||||||
};
|
};
|
||||||
enrollHost(createEnrollment(hostId).token, inventory);
|
agentCredential = enrollHost(createEnrollment(hostId).token, inventory).credential;
|
||||||
const { createLibraryImport, updateHostLibraryCaches } = await import('../../src/lib/server/game-files/index.js');
|
const { createLibraryImport, updateHostLibraryCaches } = await import('../../src/lib/server/game-files/index.js');
|
||||||
const { db } = await import('../../src/lib/server/db/index.js');
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
const libraries = ['t5', 't6'].map(game => {
|
const libraries = ['t5', 't6'].map(game => {
|
||||||
@@ -41,7 +45,8 @@ beforeEach(async () => {
|
|||||||
return { id, state: 'ready' as const };
|
return { id, state: 'ready' as const };
|
||||||
});
|
});
|
||||||
updateHostLibraryCaches(hostId, libraries.map(library => ({ libraryVersionId: library.id, status: 'ready', manifestSha256: 'a'.repeat(64) })));
|
updateHostLibraryCaches(hostId, libraries.map(library => ({ libraryVersionId: library.id, status: 'ready', manifestSha256: 'a'.repeat(64) })));
|
||||||
heartbeat(hostId, { ...inventory, libraries });
|
heartbeatInventory = { ...inventory, libraries };
|
||||||
|
heartbeat(hostId, heartbeatInventory);
|
||||||
const { T6_ZOMBIE_MAPS } = await import('../../src/lib/map-aliases.js');
|
const { T6_ZOMBIE_MAPS } = await import('../../src/lib/map-aliases.js');
|
||||||
const now = new Date().toISOString();
|
const now = new Date().toISOString();
|
||||||
db().transaction(() => {
|
db().transaction(() => {
|
||||||
@@ -68,6 +73,7 @@ beforeEach(async () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
afterEach(async () => {
|
afterEach(async () => {
|
||||||
|
releasePlacement?.();
|
||||||
await new Promise<void>(resolve => setImmediate(resolve));
|
await new Promise<void>(resolve => setImmediate(resolve));
|
||||||
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
|
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
|
||||||
await processProvisioning();
|
await processProvisioning();
|
||||||
@@ -78,7 +84,44 @@ afterEach(async () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
describe('matching a native collection', () => {
|
describe('matching a native collection', () => {
|
||||||
it('admits nine removals and ten creations without copying plugin bytes into every queued launch', async () => {
|
it('places queued creations and completes all exact removals during later worker passes', async () => {
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
|
||||||
|
saveFleetSettings({ ...fleetSettings(), createConcurrency: 10, destroyConcurrency: 9 });
|
||||||
|
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
|
||||||
|
const { previewCollection, queueCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
|
||||||
|
const { T5_ZOMBIE_MAPS } = await import('../../src/lib/map-aliases.js');
|
||||||
|
const blueprint = saveBlueprint({ name: 'Replace nine with ten', entries: T5_ZOMBIE_MAPS.filter(map => map.slug !== 'zombietron')
|
||||||
|
.map(map => ({ game: 't5', mapSlug: map.slug, count: 1 })) });
|
||||||
|
const preview = previewCollection(blueprint.id, 'exact');
|
||||||
|
const queued = queueCollection({ previewId: preview.id,
|
||||||
|
keys: Object.fromEntries(preview.servers.map(server => [server.name!, `key-${server.name}`])) });
|
||||||
|
expect(queueCollection({ previewId: preview.id })).toEqual(queued);
|
||||||
|
expect(db().prepare('SELECT id FROM native_instances').all()).toHaveLength(9);
|
||||||
|
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toHaveLength(18);
|
||||||
|
|
||||||
|
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
|
||||||
|
await processProvisioning();
|
||||||
|
const operations = db().prepare("SELECT id,kind FROM ec_host_operations WHERE kind IN ('instance.create','instance.remove')")
|
||||||
|
.all() as { id: string; kind: string }[];
|
||||||
|
expect(operations.filter(operation => operation.kind === 'instance.create')).toHaveLength(10);
|
||||||
|
const removals = operations.filter(operation => operation.kind === 'instance.remove');
|
||||||
|
expect(removals).toHaveLength(9);
|
||||||
|
const { reportHostOperationStatus } = await import('../../src/lib/server/hosts/index.js');
|
||||||
|
for (const operation of removals) reportHostOperationStatus(hostId, operation.id, { state: 'completed', result: { exists: false } });
|
||||||
|
await processProvisioning();
|
||||||
|
const active = db().prepare('SELECT id,port FROM native_instances WHERE retired_at IS NULL').all() as { id: string; port: number }[];
|
||||||
|
expect(active).toHaveLength(10);
|
||||||
|
expect(new Set(active.map(instance => instance.port)).size).toBe(10);
|
||||||
|
for (const instance of active) expect(db().prepare('SELECT protocol,port FROM ec_host_ports WHERE instance_id=? ORDER BY protocol').all(instance.id))
|
||||||
|
.toEqual([{ protocol: 'tcp', port: instance.port }, { protocol: 'udp', port: instance.port }]);
|
||||||
|
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toHaveLength(20);
|
||||||
|
expect(getCollection(queued.collectionId!)?.servers.filter(server => server.action === 'destroy').map(server => server.state))
|
||||||
|
.toEqual(Array(9).fill('done'));
|
||||||
|
expect(db().prepare("SELECT id FROM ec_host_operations WHERE kind IN ('instance.create','instance.remove')").all()).toHaveLength(19);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('accepts nine removals and ten unplaced creations before capacity checks or file loading', async () => {
|
||||||
const { db } = await import('../../src/lib/server/db/index.js');
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
const { createScript, setScope, deleteScript } = await import('../../src/lib/server/fleet/scripts.js');
|
const { createScript, setScope, deleteScript } = await import('../../src/lib/server/fleet/scripts.js');
|
||||||
const content = Buffer.alloc(8 * 1024 ** 2, 42);
|
const content = Buffer.alloc(8 * 1024 ** 2, 42);
|
||||||
@@ -91,6 +134,10 @@ describe('matching a native collection', () => {
|
|||||||
const { nativeRequest } = await import('../../src/lib/server/backends/native.js');
|
const { nativeRequest } = await import('../../src/lib/server/backends/native.js');
|
||||||
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
|
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
|
||||||
const { POST } = await import('../../src/routes/api/deployment-collections/+server.js');
|
const { POST } = await import('../../src/routes/api/deployment-collections/+server.js');
|
||||||
|
const { load } = await import('../../src/routes/collections/launches/[id]/+page.server.js');
|
||||||
|
const { GET: collectionResponse } = await import('../../src/routes/api/deployment-collections/[id]/+server.js');
|
||||||
|
const { GET: fleetResponse } = await import('../../src/routes/api/servers/+server.js');
|
||||||
|
const { POST: agentHeartbeat } = await import('../../src/routes/api/agent/heartbeat/+server.js');
|
||||||
const { T5_ZOMBIE_MAPS } = await import('../../src/lib/map-aliases.js');
|
const { T5_ZOMBIE_MAPS } = await import('../../src/lib/map-aliases.js');
|
||||||
const blueprint = saveBlueprint({ name: 'Match ten T5 lobbies', entries: T5_ZOMBIE_MAPS.filter(map => map.slug !== 'zombietron')
|
const blueprint = saveBlueprint({ name: 'Match ten T5 lobbies', entries: T5_ZOMBIE_MAPS.filter(map => map.slug !== 'zombietron')
|
||||||
.map(map => ({ game: 't5', mapSlug: map.slug, count: 1 })) });
|
.map(map => ({ game: 't5', mapSlug: map.slug, count: 1 })) });
|
||||||
@@ -102,6 +149,23 @@ describe('matching a native collection', () => {
|
|||||||
.get(preview.id) as { bytes: number };
|
.get(preview.id) as { bytes: number };
|
||||||
expect(storedPreview.bytes).toBeLessThan(100_000);
|
expect(storedPreview.bytes).toBeLessThan(100_000);
|
||||||
|
|
||||||
|
const capacity = await import('../../src/lib/server/hosts/capacity.js');
|
||||||
|
const placement = vi.spyOn(capacity, 'choosePlacement').mockImplementation(() => {
|
||||||
|
throw new Error('Placement is waiting for fresh host capacity');
|
||||||
|
});
|
||||||
|
const preparation = await import('../../src/lib/server/provisioning/launch-preparation.js');
|
||||||
|
const evaluate = preparation.evaluateNativePlacement;
|
||||||
|
let placementStarted!: () => void;
|
||||||
|
const started = new Promise<void>(resolve => { placementStarted = resolve; });
|
||||||
|
const gate = new Promise<void>(resolve => { releasePlacement = resolve; });
|
||||||
|
const evaluation = vi.spyOn(preparation, 'evaluateNativePlacement').mockImplementation(async options => {
|
||||||
|
placementStarted();
|
||||||
|
await gate;
|
||||||
|
return evaluate(options);
|
||||||
|
});
|
||||||
|
const scripts = await import('../../src/lib/server/fleet/scripts.js');
|
||||||
|
const fullVersion = vi.spyOn(scripts, 'getVersion');
|
||||||
|
|
||||||
const submit = (keys?: Record<string, string>) => POST({ request: new Request('http://localhost/api/deployment-collections', {
|
const submit = (keys?: Record<string, string>) => POST({ request: new Request('http://localhost/api/deployment-collections', {
|
||||||
method: 'POST', body: JSON.stringify({ previewId: preview.id, keys }) }) });
|
method: 'POST', body: JSON.stringify({ previewId: preview.id, keys }) }) });
|
||||||
const response = await submit(Object.fromEntries(preview.servers.map(server => [server.name!, `key-${server.name}`])));
|
const response = await submit(Object.fromEntries(preview.servers.map(server => [server.name!, `key-${server.name}`])));
|
||||||
@@ -109,21 +173,54 @@ describe('matching a native collection', () => {
|
|||||||
expect(response.status).toBe(202);
|
expect(response.status).toBe(202);
|
||||||
expect(body).toMatchObject({ ok: true, collectionId: expect.any(String) });
|
expect(body).toMatchObject({ ok: true, collectionId: expect.any(String) });
|
||||||
expect(await (await submit()).json()).toEqual(body);
|
expect(await (await submit()).json()).toEqual(body);
|
||||||
|
expect(load({ params: { id: body.collectionId } })).toEqual({});
|
||||||
// Exercise the real route and worker: admission must return before
|
// Exercise the real route and worker: admission must return before
|
||||||
// loading launch payloads or inspecting the nine servers to remove.
|
// loading launch payloads or inspecting the nine servers to remove.
|
||||||
expect(nativeRequest).not.toHaveBeenCalled();
|
expect(nativeRequest).not.toHaveBeenCalled();
|
||||||
|
expect(placement).not.toHaveBeenCalled();
|
||||||
|
expect(evaluation).not.toHaveBeenCalled();
|
||||||
|
expect(fullVersion).not.toHaveBeenCalled();
|
||||||
expect(db().prepare("SELECT id FROM ec_host_operations WHERE kind='instance.create'").all()).toEqual([]);
|
expect(db().prepare("SELECT id FROM ec_host_operations WHERE kind='instance.create'").all()).toEqual([]);
|
||||||
for (const table of ['server_launch_previews', 'server_launch_recipes']) {
|
expect(db().prepare('SELECT id FROM native_instances').all()).toHaveLength(9);
|
||||||
const rows = db().prepare(`SELECT length(recipe) AS bytes FROM ${table}`).all() as { bytes: number }[];
|
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toHaveLength(18);
|
||||||
|
expect(db().prepare('SELECT deployment_id FROM native_creation_intents').all()).toHaveLength(10);
|
||||||
|
expect(db().prepare('SELECT id FROM server_launch_previews').all()).toEqual([]);
|
||||||
|
const rows = db().prepare('SELECT length(recipe) AS bytes FROM server_launch_recipes').all() as { bytes: number }[];
|
||||||
expect(rows).toHaveLength(10);
|
expect(rows).toHaveLength(10);
|
||||||
expect(rows.every(row => row.bytes < 10_000)).toBe(true);
|
expect(rows.every(row => row.bytes < 10_000)).toBe(true);
|
||||||
}
|
|
||||||
const collection = getCollection(body.collectionId)!;
|
const collection = getCollection(body.collectionId)!;
|
||||||
expect(collection.collection).toMatchObject({ mode: 'exact', total: 19 });
|
expect(collection.collection).toMatchObject({ mode: 'exact', total: 19 });
|
||||||
expect(collection.servers.filter(server => server.action === 'destroy')).toHaveLength(9);
|
expect(collection.servers.filter(server => server.action === 'destroy')).toHaveLength(9);
|
||||||
expect(collection.servers.filter(server => server.action === 'create')).toHaveLength(10);
|
expect(collection.servers.filter(server => server.action === 'create')).toHaveLength(10);
|
||||||
|
const { nativeDeployment } = await import('../../src/lib/server/backends/native.js');
|
||||||
|
for (const server of collection.servers.filter(server => server.action === 'create')) {
|
||||||
|
expect(nativeDeployment(server.id)).toBe(true);
|
||||||
|
expect(getDeployment(server.id)?.backend).toBe('native');
|
||||||
|
}
|
||||||
|
|
||||||
await new Promise<void>(resolve => setImmediate(resolve));
|
await new Promise<void>(resolve => setImmediate(resolve));
|
||||||
|
await processProvisioning({ wait: false });
|
||||||
|
await started;
|
||||||
|
expect(evaluation).toHaveBeenCalled();
|
||||||
|
expect(placement).not.toHaveBeenCalled();
|
||||||
|
expect(db().prepare("SELECT id FROM ec_host_operations WHERE kind='instance.create'").all()).toEqual([]);
|
||||||
|
expect(getCollection(body.collectionId)?.collection.total).toBe(19);
|
||||||
|
const collectionPage = collectionResponse({ params: { id: body.collectionId } });
|
||||||
|
expect(collectionPage.status).toBe(200);
|
||||||
|
expect((await collectionPage.json()).servers).toHaveLength(19);
|
||||||
|
const fleet = fleetResponse();
|
||||||
|
expect(fleet.status).toBe(200);
|
||||||
|
expect((await fleet.json()).servers).toHaveLength(9);
|
||||||
|
const heartbeatResponse = await agentHeartbeat({ request: new Request('https://controller.test/api/agent/heartbeat', {
|
||||||
|
method: 'POST', headers: { authorization: `Bearer ${agentCredential}` }, body: JSON.stringify({ inventory: heartbeatInventory })
|
||||||
|
}) });
|
||||||
|
expect(heartbeatResponse.status).toBe(200);
|
||||||
|
expect(await heartbeatResponse.json()).toMatchObject({ host: { id: hostId, connected: true }, heartbeatSeconds: 15 });
|
||||||
|
// All three endpoints complete while the background placement is still paused.
|
||||||
|
expect(placement).not.toHaveBeenCalled();
|
||||||
|
expect(db().prepare("SELECT id FROM ec_host_operations WHERE kind='instance.create'").all()).toEqual([]);
|
||||||
|
placement.mockRestore(); fullVersion.mockRestore();
|
||||||
|
releasePlacement!();
|
||||||
await processProvisioning();
|
await processProvisioning();
|
||||||
const dispatched = db().prepare(`SELECT n.deployment_id,o.id FROM native_provisioning n
|
const dispatched = db().prepare(`SELECT n.deployment_id,o.id FROM native_provisioning n
|
||||||
JOIN ec_host_operations o ON o.id=n.operation_id WHERE o.kind='instance.create'`).all() as { deployment_id: string; id: string }[];
|
JOIN ec_host_operations o ON o.id=n.operation_id WHERE o.kind='instance.create'`).all() as { deployment_id: string; id: string }[];
|
||||||
|
|||||||
@@ -481,6 +481,7 @@ describe('server creation map selection', () => {
|
|||||||
expect(getDeployment(id)?.deployment).toMatchObject({ state: 'awaiting-health', stage: 'Waiting for RCON or fresh telemetry health: timeout' });
|
expect(getDeployment(id)?.deployment).toMatchObject({ state: 'awaiting-health', stage: 'Waiting for RCON or fresh telemetry health: timeout' });
|
||||||
for (const map of [null, 'zombie_moon']) {
|
for (const map of [null, 'zombie_moon']) {
|
||||||
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: true, state: 'running' });
|
vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: true, state: 'running' });
|
||||||
|
if (created.port === null) throw new Error('A provisioned legacy server must have a port.');
|
||||||
vi.mocked(rconStatusResult).mockResolvedValueOnce({ ok: true, endpoint: { host: 'localhost', port: created.port }, rawReply: '', status: { players: 0, map, gametype: null, roster: [] } });
|
vi.mocked(rconStatusResult).mockResolvedValueOnce({ ok: true, endpoint: { host: 'localhost', port: created.port }, rawReply: '', status: { players: 0, map, gametype: null, roster: [] } });
|
||||||
await processCreations();
|
await processCreations();
|
||||||
expect(getDeployment(id)?.deployment).toMatchObject({ state: 'awaiting-health', finished_at: null, stage: expect.stringContaining('expected zombie_cosmodrome') });
|
expect(getDeployment(id)?.deployment).toMatchObject({ state: 'awaiting-health', finished_at: null, stage: expect.stringContaining('expected zombie_cosmodrome') });
|
||||||
|
|||||||
@@ -0,0 +1,104 @@
|
|||||||
|
import { setImmediate } from 'node:timers/promises';
|
||||||
|
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||||
|
import { newFixture, type Fixture } from '../helpers/fixture.js';
|
||||||
|
|
||||||
|
let fixture: Fixture;
|
||||||
|
let hostId: string;
|
||||||
|
const GiB = 1024 ** 3;
|
||||||
|
const epoch = Date.parse('2026-10-05T12:00:00Z');
|
||||||
|
const inventory = { protocolVersion: 1, dockerRunning: true, directoriesWritable: true, storageAvailable: 100 * GiB,
|
||||||
|
cpuCount: 16, cpuUsagePercent: 0, memoryTotal: 64 * GiB, memoryAvailable: 60 * GiB,
|
||||||
|
runtimeImages: ['image'], libraries: [{ id: 't5-library', state: 'ready' }, { id: 't6-library', state: 'ready' }] };
|
||||||
|
const t5 = { libraryId: 't5-library', image: 'image', automatic: true };
|
||||||
|
const t6 = { libraryId: 't6-library', image: 'image', automatic: true };
|
||||||
|
async function settle() { await setImmediate(); await setImmediate(); }
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
vi.resetModules();
|
||||||
|
fixture = await newFixture([]);
|
||||||
|
vi.useFakeTimers({ toFake: ['Date'] }); vi.setSystemTime(epoch);
|
||||||
|
const hosts = await import('../../src/lib/server/hosts/index.js');
|
||||||
|
hostId = hosts.createHost({ name: 'Availability host', portMin: 28960, portMax: 28962 }).id;
|
||||||
|
hosts.enrollHost(hosts.createEnrollment(hostId).token, inventory);
|
||||||
|
hosts.heartbeat(hostId, inventory);
|
||||||
|
await settle();
|
||||||
|
});
|
||||||
|
afterEach(async () => { vi.restoreAllMocks(); vi.useRealTimers(); await fixture.cleanup(); });
|
||||||
|
|
||||||
|
describe('cached launch availability', () => {
|
||||||
|
it('reads cached estimates without placement, historical scans, or repeated database reads', async () => {
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const capacity = await import('../../src/lib/server/hosts/capacity.js');
|
||||||
|
const { launchAvailability } = await import('../../src/lib/server/hosts/availability.js');
|
||||||
|
const placement = vi.spyOn(capacity, 'choosePlacement');
|
||||||
|
const prepare = vi.spyOn(db(), 'prepare');
|
||||||
|
for (let count = 0; count < 100; count++) {
|
||||||
|
expect(launchAvailability([t5, t6]).availability).toMatchObject({
|
||||||
|
updatedAt: new Date(epoch).toISOString(), requested: 2, estimatedAvailable: 2, portsAvailable: 3
|
||||||
|
});
|
||||||
|
}
|
||||||
|
expect(prepare).not.toHaveBeenCalled();
|
||||||
|
expect(placement).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('shares CPU, memory and ports across games and refreshes measurements on heartbeat', async () => {
|
||||||
|
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
|
||||||
|
const { launchAvailability } = await import('../../src/lib/server/hosts/availability.js');
|
||||||
|
heartbeat(hostId, { ...inventory, memoryAvailable: GiB + 500_000_000 });
|
||||||
|
const limited = launchAvailability([t5, t6]);
|
||||||
|
expect(limited.availability).toMatchObject({ requested: 2, estimatedAvailable: 1, portsAvailable: 3,
|
||||||
|
cpuCoresAvailable: 15.2, memoryBytesAvailable: GiB });
|
||||||
|
expect(limited.warnings.join(' ')).toContain('1 of 2');
|
||||||
|
vi.setSystemTime(epoch + 15_000);
|
||||||
|
heartbeat(hostId, { ...inventory, memoryAvailable: 2 * GiB + 500_000_000 });
|
||||||
|
expect(launchAvailability([t5, t6]).availability).toMatchObject({ estimatedAvailable: 2,
|
||||||
|
memoryBytesAvailable: 2 * GiB, updatedAt: new Date(epoch + 15_000).toISOString() });
|
||||||
|
heartbeat(hostId, { ...inventory, cpuUsagePercent: 100 });
|
||||||
|
expect(launchAvailability([t5]).availability.estimatedAvailable).toBe(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('refreshes port reservations and startup queue snapshots after lifecycle events', async () => {
|
||||||
|
const { reserveHostPort, releaseHostPorts } = await import('../../src/lib/server/hosts/index.js');
|
||||||
|
const { launchAvailability } = await import('../../src/lib/server/hosts/availability.js');
|
||||||
|
const { beginDeployment, deploymentStage } = await import('../../src/lib/server/provisioning/history.js');
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const limit = launchAvailability([t5]).availability.startupLimit!;
|
||||||
|
reserveHostPort(hostId, 'reserved', { port: 28960, protocol: 'udp' });
|
||||||
|
reserveHostPort(hostId, 'reserved', { port: 28960, protocol: 'tcp' });
|
||||||
|
await settle();
|
||||||
|
expect(launchAvailability([t5, t6, t5]).availability).toMatchObject({ portsAvailable: 2, estimatedAvailable: 2 });
|
||||||
|
const active = beginDeployment('t5_zombie_theater_1', 'create');
|
||||||
|
const queued = beginDeployment('t6_zombie_tranzit_1', 'create');
|
||||||
|
db().prepare("UPDATE server_deployments SET state='queued' WHERE id=?").run(queued);
|
||||||
|
deploymentStage(queued, 'Awaiting placement', { checkpoint: { key: 'queued', label: 'Queued for placement' } });
|
||||||
|
await settle();
|
||||||
|
expect(launchAvailability([t5]).availability).toMatchObject({ queued: 1, startupSlots: Math.max(0, limit - 1) });
|
||||||
|
db().prepare("UPDATE server_deployments SET state='done' WHERE id=?").run(active);
|
||||||
|
releaseHostPorts('reserved');
|
||||||
|
await settle();
|
||||||
|
expect(launchAvailability([t5]).availability).toMatchObject({ portsAvailable: 3, startupSlots: limit });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('stops counting a stale host without presenting old measurements as current capacity', async () => {
|
||||||
|
const { launchAvailability } = await import('../../src/lib/server/hosts/availability.js');
|
||||||
|
vi.setSystemTime(epoch + 91_000);
|
||||||
|
const result = launchAvailability([t5]);
|
||||||
|
expect(result.availability).toMatchObject({ updatedAt: new Date(epoch).toISOString(), estimatedAvailable: 0, portsAvailable: 0 });
|
||||||
|
expect(result.warnings.join(' ')).toContain('Waiting for an eligible host');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('keeps heartbeats working while provisioning schema is unavailable and refreshes after initialization', async () => {
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
|
||||||
|
const { launchAvailability } = await import('../../src/lib/server/hosts/availability.js');
|
||||||
|
const errors = vi.spyOn(console, 'error');
|
||||||
|
db().exec('ALTER TABLE native_provisioning RENAME TO unavailable_native_provisioning');
|
||||||
|
expect(() => heartbeat(hostId, inventory)).not.toThrow();
|
||||||
|
expect(launchAvailability([t5]).availability.updatedAt).toBeNull();
|
||||||
|
await settle();
|
||||||
|
expect(errors).not.toHaveBeenCalled();
|
||||||
|
db().exec('ALTER TABLE unavailable_native_provisioning RENAME TO native_provisioning');
|
||||||
|
heartbeat(hostId, inventory);
|
||||||
|
expect(launchAvailability([t5]).availability).toMatchObject({ estimatedAvailable: 1, portsAvailable: 3 });
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -77,6 +77,9 @@ describe('host deletion', () => {
|
|||||||
const queued = hosts.deleteHost(hosts.host.id);
|
const queued = hosts.deleteHost(hosts.host.id);
|
||||||
expect(queued.deleted).toBe(false);
|
expect(queued.deleted).toBe(false);
|
||||||
expect(hosts.getHost(hosts.host.id)?.decommission?.stage).toBe('queued');
|
expect(hosts.getHost(hosts.host.id)?.decommission?.stage).toBe('queued');
|
||||||
|
const { nativeLaunchSelection } = await import('../../src/lib/server/provisioning/native.js');
|
||||||
|
expect(() => nativeLaunchSelection({ hostId: hosts.host.id, libraryId: 'version', image: 'plutainer:test' }, 't5'))
|
||||||
|
.toThrow('The selected host is being removed');
|
||||||
expect(hosts.authenticateCredential(hosts.credentials.credential)?.id).toBe(hosts.host.id);
|
expect(hosts.authenticateCredential(hosts.credentials.credential)?.id).toBe(hosts.host.id);
|
||||||
await finishCleanup(hosts);
|
await finishCleanup(hosts);
|
||||||
|
|
||||||
@@ -120,6 +123,44 @@ describe('host deletion', () => {
|
|||||||
expect(hosts.authenticateCredential(hosts.credentials.credential)?.id).toBe(hosts.host.id);
|
expect(hosts.authenticateCredential(hosts.credentials.credential)?.id).toBe(hosts.host.id);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('blocks an accepted unplaced pinned launch until it is cancelled', async () => {
|
||||||
|
const hosts = await enrolled();
|
||||||
|
const { queueNativeCreationIntent } = await import('../../src/lib/server/provisioning/native.js');
|
||||||
|
const { cancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const queued = queueNativeCreationIntent({ server: 't5_zombie_theater_1', game: 't5', mapSlug: 'theater',
|
||||||
|
plutoKey: 'saved-key', selection: { hostId: hosts.host.id, libraryId: 'library', image: 'plutainer:test' } });
|
||||||
|
expect(queued.ok).toBe(true);
|
||||||
|
if (!queued.ok) throw new Error(queued.message);
|
||||||
|
expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]);
|
||||||
|
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
|
||||||
|
expect(hosts.hostDeletionStatus(hosts.host.id)).toEqual({ allowed: false,
|
||||||
|
blockers: [expect.stringContaining('Cancel the queued server launches pinned to this host')] });
|
||||||
|
expect(() => hosts.deleteHost(hosts.host.id)).toThrow(expect.objectContaining({ status: 409 }));
|
||||||
|
expect(hosts.getHost(hosts.host.id)?.decommission).toBeNull();
|
||||||
|
expect(cancelCreation(queued.deploymentId!)).toBeTruthy();
|
||||||
|
expect(hosts.hostDeletionStatus(hosts.host.id)).toEqual({ allowed: true, blockers: [] });
|
||||||
|
await finishCleanup(hosts);
|
||||||
|
expect(hosts.getHost(hosts.host.id)).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('does not pin a host for automatic launches or launches assigned elsewhere', async () => {
|
||||||
|
const hosts = await enrolled();
|
||||||
|
const other = hosts.createHost({ name: 'Other host' });
|
||||||
|
const { queueNativeCreationIntent } = await import('../../src/lib/server/provisioning/native.js');
|
||||||
|
for (const [index, selection] of [
|
||||||
|
{ hostId: hosts.host.id, automatic: true },
|
||||||
|
{ hostId: other.id, automatic: false }
|
||||||
|
].entries()) {
|
||||||
|
expect(queueNativeCreationIntent({ server: `t5_zombie_theater_${index + 1}`, game: 't5', mapSlug: 'theater',
|
||||||
|
plutoKey: 'saved-key', selection: { ...selection, libraryId: 'library', image: 'plutainer:test' } }).ok).toBe(true);
|
||||||
|
}
|
||||||
|
expect(hosts.hostDeletionStatus(hosts.host.id)).toEqual({ allowed: true, blockers: [] });
|
||||||
|
expect(hosts.hostDeletionStatus(other.id).allowed).toBe(false);
|
||||||
|
await finishCleanup(hosts);
|
||||||
|
expect(hosts.getHost(hosts.host.id)).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
it('blocks residual port reservations independently of live instances', async () => {
|
it('blocks residual port reservations independently of live instances', async () => {
|
||||||
const hosts = await enrolled();
|
const hosts = await enrolled();
|
||||||
hosts.reserveHostPort(hosts.host.id, 'unresolved-instance');
|
hosts.reserveHostPort(hosts.host.id, 'unresolved-instance');
|
||||||
|
|||||||
@@ -243,6 +243,68 @@ describe('automatic host preparation', () => {
|
|||||||
expect(libraries.requireReadyLibrary(id, old.id).id).toBe(old.id);
|
expect(libraries.requireReadyLibrary(id, old.id).id).toBe(old.id);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it.each([false, true])('prepares queued runtime and library versions after newer publication (automatic placement: %s)', async automatic => {
|
||||||
|
const old = await publish();
|
||||||
|
const id = enroll({ runtimeImages: [runtime] });
|
||||||
|
const other = enroll({ runtimeImages: [runtime] });
|
||||||
|
const image = 'ghcr.io/example/plutainer:queued-version';
|
||||||
|
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
|
||||||
|
for (let index = 0; index < 2; index++) {
|
||||||
|
expect(queueNativeCreation({ game: 't5', gameMode: 'zombie', mapSlug: 'theater', libraryId: old.id,
|
||||||
|
hostId: automatic ? undefined : id, image, plutoKey: 'saved-key' }).ok).toBe(true);
|
||||||
|
}
|
||||||
|
const latest = await publish();
|
||||||
|
const status = preparation.hostPreparationStatus(hosts.getHost(id)!);
|
||||||
|
expect(status.items.filter(item => item.label === image)).toHaveLength(1);
|
||||||
|
expect(status.items.filter(item => item.label.includes('Queued version'))).toHaveLength(1);
|
||||||
|
const otherStatus = preparation.hostPreparationStatus(hosts.getHost(other)!);
|
||||||
|
expect(otherStatus.items.some(item => item.label === image)).toBe(automatic);
|
||||||
|
expect(otherStatus.items.some(item => item.label.includes('Queued version'))).toBe(automatic);
|
||||||
|
expect(operations(id)).toEqual([]);
|
||||||
|
|
||||||
|
preparation.reconcileHostPreparation(id);
|
||||||
|
const pull = operations(id)[0];
|
||||||
|
expect(pull).toMatchObject({ kind: 'runtime.pull', payload: { image } });
|
||||||
|
vi.setSystemTime(Date.now() + 1);
|
||||||
|
hosts.reportHostOperation(id, pull.id, { state: 'completed' });
|
||||||
|
heartbeat(id, { runtimeImages: [runtime, image] });
|
||||||
|
preparation.reconcileHostPreparation(id);
|
||||||
|
const download = operations(id)[0];
|
||||||
|
expect(download).toMatchObject({ kind: 'library.download', payload: { libraryVersionId: old.id, automatic: true } });
|
||||||
|
preparation.reconcileHostPreparation(id);
|
||||||
|
expect(hosts.getHostOperation(download.id)?.state).toBe('pending');
|
||||||
|
expect(operations(id)).toHaveLength(2);
|
||||||
|
|
||||||
|
verified(id, old);
|
||||||
|
vi.setSystemTime(Date.now() + 1);
|
||||||
|
hosts.reportHostOperation(id, download.id, { state: 'completed' });
|
||||||
|
heartbeat(id, { runtimeImages: [runtime, image], libraries: [{ id: old.id, state: 'ready' }] });
|
||||||
|
preparation.reconcileHostPreparation(id);
|
||||||
|
expect(operations(id)[0]).toMatchObject({ kind: 'library.download', payload: { libraryVersionId: latest.id } });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('retains an undelivered library download for an accepted intent until that intent is cancelled', async () => {
|
||||||
|
const old = await publish();
|
||||||
|
const id = enroll({ runtimeImages: [runtime] });
|
||||||
|
preparation.reconcileHostPreparation(id);
|
||||||
|
const download = operations(id)[0];
|
||||||
|
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
|
||||||
|
const launch = queueNativeCreation({ game: 't5', gameMode: 'zombie', mapSlug: 'theater', libraryId: old.id,
|
||||||
|
hostId: id, image: runtime, plutoKey: 'saved-key' });
|
||||||
|
expect(launch.ok).toBe(true);
|
||||||
|
if (!launch.ok) throw new Error(launch.message);
|
||||||
|
const latest = await publish();
|
||||||
|
preparation.reconcileHostPreparation(id);
|
||||||
|
expect(hosts.getHostOperation(download.id)?.state).toBe('pending');
|
||||||
|
expect(operations(id)).toHaveLength(1);
|
||||||
|
|
||||||
|
const { cancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
|
||||||
|
expect(cancelCreation(launch.deploymentId)).toEqual(expect.any(String));
|
||||||
|
preparation.reconcileHostPreparation(id);
|
||||||
|
expect(hosts.getHostOperation(download.id)).toMatchObject({ state: 'failed', error: expect.stringContaining('Superseded') });
|
||||||
|
expect(operations(id)[0]).toMatchObject({ kind: 'library.download', payload: { libraryVersionId: latest.id } });
|
||||||
|
});
|
||||||
|
|
||||||
it('finishes a delivered download before following a newer upload under a new operation ID', async () => {
|
it('finishes a delivered download before following a newer upload under a new operation ID', async () => {
|
||||||
const old = await publish();
|
const old = await publish();
|
||||||
const id = enroll({ runtimeImages: [runtime] });
|
const id = enroll({ runtimeImages: [runtime] });
|
||||||
|
|||||||
@@ -0,0 +1,197 @@
|
|||||||
|
import Database from 'better-sqlite3';
|
||||||
|
import { createHash } from 'node:crypto';
|
||||||
|
import { join } from 'node:path';
|
||||||
|
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||||
|
import schema from '../../src/lib/server/db/schema.sql?raw';
|
||||||
|
import { newFixture, type Fixture } from '../helpers/fixture.js';
|
||||||
|
|
||||||
|
let fixture: Fixture;
|
||||||
|
const server = 't5_zombie_theater_1';
|
||||||
|
beforeEach(async () => {
|
||||||
|
vi.resetModules();
|
||||||
|
fixture = await newFixture([]);
|
||||||
|
vi.stubEnv('KOMODO_ADDRESS', ''); vi.stubEnv('KOMODO_API_KEY', ''); vi.stubEnv('KOMODO_API_SECRET', '');
|
||||||
|
vi.stubEnv('RCON_PASSWORD', '');
|
||||||
|
});
|
||||||
|
afterEach(async () => { vi.restoreAllMocks(); await fixture.cleanup(); vi.unstubAllEnvs(); });
|
||||||
|
|
||||||
|
async function intent() {
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const { ensureHostSchema } = await import('../../src/lib/server/hosts/schema.js');
|
||||||
|
ensureHostSchema(db());
|
||||||
|
const { beginDeployment } = await import('../../src/lib/server/provisioning/history.js');
|
||||||
|
const id = beginDeployment(server, 'create');
|
||||||
|
db().prepare("UPDATE server_deployments SET state='queued' WHERE id=?").run(id);
|
||||||
|
db().prepare('INSERT INTO native_creation_intents(deployment_id,selection) VALUES(?,?)')
|
||||||
|
.run(id, JSON.stringify({ automatic: true, libraryId: 'library', image: 'plutainer:test' }));
|
||||||
|
db().prepare(`INSERT INTO server_creation_jobs(deployment_id,server,port,game_mode,map_slug,expected_map)
|
||||||
|
VALUES(?,?,NULL,'zombie','theater','zombie_theater')`).run(id, server);
|
||||||
|
return id;
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('native creation intents before placement', () => {
|
||||||
|
it('retains its identity and native history across restart without falling into legacy provisioning', async () => {
|
||||||
|
const id = await intent();
|
||||||
|
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const { getDeployment, interruptDeployments } = await import('../../src/lib/server/provisioning/history.js');
|
||||||
|
const { nativeDeployment, unplacedNativeCreation } = await import('../../src/lib/server/backends/native.js');
|
||||||
|
const { nextLobbyNumber, processCreations } = await import('../../src/lib/server/provisioning/instances.js');
|
||||||
|
const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js');
|
||||||
|
expect(nativeDeployment(id)).toBe(true);
|
||||||
|
expect(unplacedNativeCreation(server)).toBe(id);
|
||||||
|
expect(nextLobbyNumber('t5_zombie_theater')).toBe(2);
|
||||||
|
expect(requestLifecycle(server, 'stop')).toMatchObject({ ok: false, status: 409 });
|
||||||
|
closeDb();
|
||||||
|
interruptDeployments();
|
||||||
|
await processCreations();
|
||||||
|
expect(getDeployment(id)).toMatchObject({ backend: 'native', canCancelCreation: true, agentOperation: null,
|
||||||
|
deployment: { state: 'queued', server_exists: 0 } });
|
||||||
|
expect(db().prepare('SELECT port FROM server_creation_jobs WHERE deployment_id=?').get(id)).toEqual({ port: null });
|
||||||
|
expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]);
|
||||||
|
expect(db().prepare('SELECT deployment_id FROM native_provisioning').all()).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('cancels unplaced work locally once, retains keys, and frees its logical server name', async () => {
|
||||||
|
const id = await intent();
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const { saveServerKey, getServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
|
||||||
|
const { cancelCreation, canCancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
|
||||||
|
const { getDeployment } = await import('../../src/lib/server/provisioning/history.js');
|
||||||
|
const { nativeDeployment, unplacedNativeCreation } = await import('../../src/lib/server/backends/native.js');
|
||||||
|
const { nextLobbyNumber } = await import('../../src/lib/server/provisioning/instances.js');
|
||||||
|
saveServerKey(server, 'saved-key');
|
||||||
|
expect(canCancelCreation(id)).toBe(true);
|
||||||
|
const cleanup = cancelCreation(id)!;
|
||||||
|
expect(cleanup).toEqual(expect.any(String));
|
||||||
|
expect(cancelCreation(id)).toBe(cleanup);
|
||||||
|
expect(canCancelCreation(id)).toBe(false);
|
||||||
|
expect(nativeDeployment(cleanup)).toBe(true);
|
||||||
|
expect(getDeployment(id)).toMatchObject({ backend: 'native', cancellation: { state: 'done' }, deployment: { state: 'cancelled' } });
|
||||||
|
expect(getDeployment(cleanup)).toMatchObject({ backend: 'native', deployment: { state: 'done', server_deleted_at: expect.any(String) } });
|
||||||
|
expect(unplacedNativeCreation(server)).toBeNull();
|
||||||
|
expect(nextLobbyNumber('t5_zombie_theater')).toBe(1);
|
||||||
|
expect(getServerKey(server)).toBe('saved-key');
|
||||||
|
expect(db().prepare('SELECT id FROM ec_host_operations').all()).toEqual([]);
|
||||||
|
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('will not cancel an unplaced historical intent against a replacement server', async () => {
|
||||||
|
const id = await intent();
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
db().prepare("INSERT INTO servers(name,komodo_name) VALUES(?,'replacement')").run(server);
|
||||||
|
const { cancelCreation, canCancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
|
||||||
|
expect(canCancelCreation(id)).toBe(false);
|
||||||
|
expect(cancelCreation(id)).toBeNull();
|
||||||
|
expect(db().prepare('SELECT name FROM servers WHERE name=?').get(server)).toEqual({ name: server });
|
||||||
|
expect(db().prepare('SELECT cleanup_id FROM native_creation_cancellations').all()).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it.each([false, true])('does not adopt a replacement instance during placement (previously bound: %s)', async previouslyBound => {
|
||||||
|
const id = await intent();
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const { createHost } = await import('../../src/lib/server/hosts/index.js');
|
||||||
|
const { captureRecipePlan, recipeStorage } = await import('../../src/lib/server/fleet/launch-recipes.js');
|
||||||
|
const hostId = createHost({ name: 'Replacement host' }).id;
|
||||||
|
const insert = db().prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at,retired_at)
|
||||||
|
VALUES(?,?,?,'library','plutainer:test',28960,?,?)`);
|
||||||
|
if (previouslyBound) {
|
||||||
|
insert.run('original-instance', server, hostId, '2025-01-01', '2025-01-02');
|
||||||
|
db().prepare('INSERT INTO native_provisioning(deployment_id,instance_id,generation) VALUES(?,?,1)').run(id, 'original-instance');
|
||||||
|
}
|
||||||
|
insert.run('replacement-instance', server, hostId, '2025-01-02', null);
|
||||||
|
const before = db().prepare('SELECT * FROM native_instances ORDER BY id').all();
|
||||||
|
const stored = recipeStorage(captureRecipePlan('t5', 'theater'));
|
||||||
|
db().prepare('INSERT INTO server_launch_recipes(deployment_id,recipe,recipe_summary) VALUES(?,?,?)')
|
||||||
|
.run(id, stored.recipe, stored.recipeSummary);
|
||||||
|
const preparation = await import('../../src/lib/server/provisioning/launch-preparation.js');
|
||||||
|
const placement = vi.spyOn(preparation, 'evaluateNativePlacement').mockRejectedValue(new Error('Unexpected placement'));
|
||||||
|
const { processNativeProvisioning } = await import('../../src/lib/server/provisioning/native.js');
|
||||||
|
await processNativeProvisioning(() => null);
|
||||||
|
expect(placement).not.toHaveBeenCalled();
|
||||||
|
expect(db().prepare('SELECT state,error FROM server_deployments WHERE id=?').get(id))
|
||||||
|
.toEqual({ state: 'failed', error: 'Original instance was retired or replaced; no command was sent.' });
|
||||||
|
expect(db().prepare('SELECT * FROM native_instances ORDER BY id').all()).toEqual(before);
|
||||||
|
expect(db().prepare('SELECT id FROM ec_host_operations').all()).toEqual([]);
|
||||||
|
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
|
||||||
|
expect(db().prepare('SELECT port FROM server_creation_jobs WHERE deployment_id=?').get(id)).toEqual({ port: null });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('starts work on a ready host when an older pinned intent is waiting for an offline host', async () => {
|
||||||
|
vi.stubEnv('GAME_FILES_ROOT', join(fixture.root, 'libraries'));
|
||||||
|
const image = 'plutainer:test';
|
||||||
|
const hosts = await import('../../src/lib/server/hosts/index.js');
|
||||||
|
const offline = hosts.createHost({ name: 'Offline host' });
|
||||||
|
const ready = hosts.createHost({ name: 'Ready host' });
|
||||||
|
const inventory = { protocolVersion: 1, dockerRunning: true, directoriesWritable: true,
|
||||||
|
cpuCount: 16, cpuUsagePercent: 5, memoryTotal: 32 * 1024 ** 3, memoryAvailable: 30 * 1024 ** 3,
|
||||||
|
storageAvailable: 100_000_000_000, runtimeImages: [image] };
|
||||||
|
hosts.enrollHost(hosts.createEnrollment(ready.id).token, inventory);
|
||||||
|
const libraries = await import('../../src/lib/server/game-files/index.js');
|
||||||
|
const imported = libraries.createLibraryImport({ hostId: ready.id, game: 't5' });
|
||||||
|
const bytes = Buffer.from('verified base game');
|
||||||
|
const file = { path: 'zone/base.ff', size: bytes.length, sha256: createHash('sha256').update(bytes).digest('hex') };
|
||||||
|
libraries.setImportManifest(ready.id, imported.id, [file]);
|
||||||
|
await libraries.uploadImportFile(ready.id, imported.id, file.path, new Response(bytes).body);
|
||||||
|
const library = await libraries.publishImport(ready.id, imported.id);
|
||||||
|
libraries.updateHostLibraryCaches(ready.id, [{ libraryVersionId: library.id, status: 'ready', manifestSha256: library.manifestSha256 }]);
|
||||||
|
hosts.heartbeat(ready.id, { ...inventory, libraries: [{ id: library.id, state: 'ready' }] });
|
||||||
|
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
|
||||||
|
saveFleetSettings({ ...fleetSettings(), createConcurrency: 1 });
|
||||||
|
const { queueNativeCreation, processNativeProvisioning } = await import('../../src/lib/server/provisioning/native.js');
|
||||||
|
const input = { game: 't5' as const, gameMode: 'zombie' as const, mapSlug: 'theater', libraryId: library.id, image, plutoKey: 'saved-key' };
|
||||||
|
const blocked = queueNativeCreation({ ...input, hostId: offline.id });
|
||||||
|
const available = queueNativeCreation({ ...input, hostId: ready.id });
|
||||||
|
expect(blocked.ok && available.ok).toBe(true);
|
||||||
|
if (!blocked.ok || !available.ok) throw new Error('Could not queue launch fixtures.');
|
||||||
|
await processNativeProvisioning(() => null);
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
expect(db().prepare('SELECT state FROM server_deployments WHERE id=?').get(blocked.deploymentId)).toEqual({ state: 'queued' });
|
||||||
|
expect(db().prepare('SELECT state FROM server_deployments WHERE id=?').get(available.deploymentId)).toEqual({ state: 'running' });
|
||||||
|
expect(db().prepare("SELECT host_id FROM ec_host_operations WHERE kind='instance.create'").all()).toEqual([{ host_id: ready.id }]);
|
||||||
|
expect(db().prepare('SELECT server,host_id FROM native_instances WHERE retired_at IS NULL').all())
|
||||||
|
.toEqual([{ server: available.name, host_id: ready.id }]);
|
||||||
|
expect(db().prepare('SELECT port FROM server_creation_jobs WHERE deployment_id=?').get(blocked.deploymentId)).toEqual({ port: null });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('handles exact-match removal of an unplaced launch as local cancellation', async () => {
|
||||||
|
const id = await intent();
|
||||||
|
const { destroyServer } = await import('../../src/lib/server/provisioning/instances.js');
|
||||||
|
const { getDeployment } = await import('../../src/lib/server/provisioning/history.js');
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const removal = destroyServer(server, 'blueprint-exact');
|
||||||
|
expect(removal).toMatchObject({ ok: true, deploymentId: expect.any(String) });
|
||||||
|
expect(getDeployment(id)?.deployment.state).toBe('cancelled');
|
||||||
|
expect(getDeployment(removal.deploymentId!)?.deployment.state).toBe('done');
|
||||||
|
expect(db().prepare('SELECT id FROM ec_host_operations').all()).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('migrates existing port reservations without losing rows, references, indexes, or triggers', async () => {
|
||||||
|
const legacy = new Database(fixture.dbPath);
|
||||||
|
try {
|
||||||
|
legacy.exec(schema.replace(/CREATE TABLE IF NOT EXISTS server_creation_jobs \([\s\S]*?\n\);/,
|
||||||
|
table => table.replace(' port INTEGER,', ' port INTEGER NOT NULL,')));
|
||||||
|
legacy.exec(`INSERT INTO server_deployments(id,server,action,state,stage,started_at,triggered_by)
|
||||||
|
VALUES('existing','old-server','create','queued','Queued','2025-01-01','fixture');
|
||||||
|
INSERT INTO server_creation_jobs(rowid,deployment_id,server,port,game_mode,map_slug,expected_map)
|
||||||
|
VALUES(8,'existing','old-server',28960,'zombie','theater','zombie_theater');
|
||||||
|
CREATE INDEX fixture_creation_port ON server_creation_jobs(port);
|
||||||
|
CREATE TABLE fixture_creation_reference(id TEXT REFERENCES server_creation_jobs(deployment_id));
|
||||||
|
INSERT INTO fixture_creation_reference(id) VALUES('existing');
|
||||||
|
CREATE TABLE fixture_creation_audit(port INTEGER);
|
||||||
|
CREATE TRIGGER fixture_creation_update AFTER UPDATE OF port ON server_creation_jobs
|
||||||
|
BEGIN INSERT INTO fixture_creation_audit(port) VALUES(NEW.port); END;`);
|
||||||
|
} finally { legacy.close(); }
|
||||||
|
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||||
|
for (let open = 0; open < 2; open++) {
|
||||||
|
const handle = db();
|
||||||
|
expect(handle.prepare('SELECT rowid,deployment_id,port FROM server_creation_jobs').get())
|
||||||
|
.toEqual({ rowid: 8, deployment_id: 'existing', port: 28960 });
|
||||||
|
expect(handle.prepare('SELECT id FROM fixture_creation_reference').get()).toEqual({ id: 'existing' });
|
||||||
|
expect(handle.prepare("SELECT name FROM sqlite_master WHERE name IN ('fixture_creation_port','fixture_creation_update')").all()).toHaveLength(2);
|
||||||
|
expect(handle.pragma('foreign_key_check')).toEqual([]);
|
||||||
|
closeDb();
|
||||||
|
}
|
||||||
|
db().prepare('UPDATE server_creation_jobs SET port=NULL WHERE deployment_id=?').run('existing');
|
||||||
|
expect(db().prepare('SELECT port FROM fixture_creation_audit').all()).toEqual([{ port: null }]);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -83,6 +83,33 @@ async function pausePreparation() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
describe('asynchronous native launch preparation', () => {
|
describe('asynchronous native launch preparation', () => {
|
||||||
|
it('does not place an unsubmitted intent cancelled while host ranking is pending', async () => {
|
||||||
|
const job = await queue();
|
||||||
|
const preparation = await import('../../src/lib/server/provisioning/launch-preparation.js');
|
||||||
|
const evaluate = preparation.evaluateNativePlacement;
|
||||||
|
let started!: () => void;
|
||||||
|
let release!: () => void;
|
||||||
|
const ready = new Promise<void>(resolve => { started = resolve; });
|
||||||
|
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||||
|
releases.push(release);
|
||||||
|
vi.spyOn(preparation, 'evaluateNativePlacement').mockImplementationOnce(async input => {
|
||||||
|
const result = await evaluate(input);
|
||||||
|
started(); await gate;
|
||||||
|
return result;
|
||||||
|
});
|
||||||
|
const pass = runPass();
|
||||||
|
await ready;
|
||||||
|
const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
|
||||||
|
expect(nativeInstance(job.name)).toBeNull();
|
||||||
|
const { cancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
|
||||||
|
expect(cancelCreation(job.deploymentId)).toEqual(expect.any(String));
|
||||||
|
release(); await pass;
|
||||||
|
expect(nativeInstance(job.name)).toBeNull();
|
||||||
|
expect(await operationFor(job.deploymentId)).toBeUndefined();
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
it('keeps controller timers and launch details responsive while preparing multiple large plugins', async () => {
|
it('keeps controller timers and launch details responsive while preparing multiple large plugins', async () => {
|
||||||
const { createScript } = await import('../../src/lib/server/fleet/scripts.js');
|
const { createScript } = await import('../../src/lib/server/fleet/scripts.js');
|
||||||
const { db } = await import('../../src/lib/server/db/index.js');
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
|||||||
@@ -43,10 +43,11 @@ async function prepareLibrary(game: 't5' | 't6' = 't5') {
|
|||||||
const { heartbeat } = await import('../../src/lib/server/hosts/index.js'); heartbeat(hostId, inventory());
|
const { heartbeat } = await import('../../src/lib/server/hosts/index.js'); heartbeat(hostId, inventory());
|
||||||
return ready;
|
return ready;
|
||||||
}
|
}
|
||||||
async function queue(game: 't5' | 't6' = 't5') {
|
async function queue(game: 't5' | 't6' = 't5', placement: 'pinned' | 'automatic' = 'pinned') {
|
||||||
await prepareLibrary(game); remote.map = game === 't5' ? 'zombie_theater' : 'zm_transit';
|
await prepareLibrary(game); remote.map = game === 't5' ? 'zombie_theater' : 'zm_transit';
|
||||||
const { queueCreateServer } = await import('../../src/lib/server/provisioning/instances.js');
|
const { queueCreateServer } = await import('../../src/lib/server/provisioning/instances.js');
|
||||||
const result = queueCreateServer({ game, gameMode: 'zombie', mapSlug: game === 't5' ? 'theater' : 'town', hostId, libraryId, image, plutoKey: 'saved-key' });
|
const result = queueCreateServer({ game, gameMode: 'zombie', mapSlug: game === 't5' ? 'theater' : 'town',
|
||||||
|
hostId: placement === 'pinned' ? hostId : undefined, libraryId, image, plutoKey: 'saved-key' });
|
||||||
if (!result.ok) throw new Error(result.message);
|
if (!result.ok) throw new Error(result.message);
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
@@ -73,6 +74,44 @@ async function detail(id: string) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
describe('native instance provisioning', () => {
|
describe('native instance provisioning', () => {
|
||||||
|
it('recognizes an unplaced native intent and resumes it once after a controller restart', async () => {
|
||||||
|
const job = await queue();
|
||||||
|
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const { nativeDeployment, nativeInstance } = await import('../../src/lib/server/backends/native.js');
|
||||||
|
const { getDeployment } = await import('../../src/lib/server/provisioning/history.js');
|
||||||
|
expect(job.port).toBeNull();
|
||||||
|
expect(nativeDeployment(job.deploymentId)).toBe(true);
|
||||||
|
expect(nativeInstance(job.name)).toBeNull();
|
||||||
|
expect(getDeployment(job.deploymentId)).toMatchObject({ backend: 'native', canCancelCreation: true });
|
||||||
|
expect(db().prepare('SELECT deployment_id FROM native_creation_intents').all()).toEqual([{ deployment_id: job.deploymentId }]);
|
||||||
|
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
|
||||||
|
|
||||||
|
closeDb(); vi.resetModules();
|
||||||
|
const backend = await import('../../src/lib/server/backends/native.js');
|
||||||
|
expect(backend.nativeDeployment(job.deploymentId)).toBe(true);
|
||||||
|
await advance();
|
||||||
|
const original = await operationFor(job.deploymentId);
|
||||||
|
expect(original).toMatchObject({ kind: 'instance.create', payload: { name: job.name, port: 28960 } });
|
||||||
|
await advance();
|
||||||
|
expect((await operationFor(job.deploymentId))?.id).toBe(original!.id);
|
||||||
|
const restartedDb = (await import('../../src/lib/server/db/index.js')).db();
|
||||||
|
expect(restartedDb.prepare("SELECT id FROM ec_host_operations WHERE kind='instance.create'").all()).toHaveLength(1);
|
||||||
|
expect(restartedDb.prepare('SELECT protocol FROM ec_host_ports ORDER BY protocol').all()).toEqual([{ protocol: 'tcp' }, { protocol: 'udp' }]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('still rejects missing keys and stale lobby identities before creating an intent', async () => {
|
||||||
|
await prepareLibrary();
|
||||||
|
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
|
||||||
|
const input = { game: 't5' as const, gameMode: 'zombie' as const, mapSlug: 'theater', hostId, libraryId, image };
|
||||||
|
expect(queueNativeCreation(input)).toMatchObject({ ok: false, message: expect.stringContaining('server key') });
|
||||||
|
expect(queueNativeCreation({ ...input, plutoKey: 'saved-key', expectedName: 't5_zombie_theater_2' }))
|
||||||
|
.toMatchObject({ ok: false });
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
expect(db().prepare('SELECT deployment_id FROM native_creation_intents').all()).toEqual([]);
|
||||||
|
expect(db().prepare('SELECT id FROM server_deployments').all()).toEqual([]);
|
||||||
|
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
it('accepts cancellation through the provisioning API and rejects an already completed creation', async () => {
|
it('accepts cancellation through the provisioning API and rejects an already completed creation', async () => {
|
||||||
const job = await queue();
|
const job = await queue();
|
||||||
const { POST } = await import('../../src/routes/api/provisioning/[id]/cancel/+server.js');
|
const { POST } = await import('../../src/routes/api/provisioning/[id]/cancel/+server.js');
|
||||||
@@ -341,11 +380,30 @@ describe('native instance provisioning', () => {
|
|||||||
expect(db().prepare('SELECT COUNT(*) AS count FROM ec_host_ports').get()).toEqual({ count: 0 });
|
expect(db().prepare('SELECT COUNT(*) AS count FROM ec_host_ports').get()).toEqual({ count: 0 });
|
||||||
});
|
});
|
||||||
|
|
||||||
it('keeps a queued server pinned to its accepted files when newer game files arrive', async () => {
|
it.each(['pinned', 'automatic'] as const)('keeps an unplaced %s launch pinned to its accepted files during cleanup', async placement => {
|
||||||
const job = await queue();
|
const job = await queue('t5', placement);
|
||||||
const acceptedId = libraryId;
|
const acceptedId = libraryId;
|
||||||
const latest = await prepareLibrary('t5');
|
const latest = await prepareLibrary('t5');
|
||||||
expect(latest.id).not.toBe(acceptedId);
|
expect(latest.id).not.toBe(acceptedId);
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
|
||||||
|
const { reconcileHostLibraryCleanup, cleanupControllerLibraries, getLibraryVersion, updateHostLibraryCaches }
|
||||||
|
= await import('../../src/lib/server/game-files/index.js');
|
||||||
|
const accepted = getLibraryVersion(acceptedId)!;
|
||||||
|
heartbeat(hostId, { ...inventory(), capabilities: ['library.remove'] });
|
||||||
|
expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]);
|
||||||
|
reconcileHostLibraryCleanup(hostId);
|
||||||
|
expect(db().prepare("SELECT id FROM ec_host_operations WHERE kind='library.remove' AND json_extract(payload,'$.libraryVersionId')=?")
|
||||||
|
.all(acceptedId)).toEqual([]);
|
||||||
|
|
||||||
|
// The unplaced intent also protects the controller's copy when every
|
||||||
|
// host reports it absent; its accepted version remains downloadable.
|
||||||
|
updateHostLibraryCaches(hostId, [{ libraryVersionId: acceptedId, status: 'missing' }]);
|
||||||
|
await cleanupControllerLibraries();
|
||||||
|
expect(db().prepare('SELECT version_id FROM game_file_cleanup WHERE version_id=?').get(acceptedId)).toBeUndefined();
|
||||||
|
const { stat } = await import('node:fs/promises');
|
||||||
|
expect((await stat(join(fixture.root, 'libraries', 't5', acceptedId, 'zone/base.ff'))).isFile()).toBe(true);
|
||||||
|
updateHostLibraryCaches(hostId, [{ libraryVersionId: acceptedId, status: 'ready', manifestSha256: accepted.manifestSha256 }]);
|
||||||
await advance();
|
await advance();
|
||||||
expect((await operationFor(job.deploymentId))?.payload.libraryId).toBe(acceptedId);
|
expect((await operationFor(job.deploymentId))?.payload.libraryId).toBe(acceptedId);
|
||||||
const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
|
const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
|
||||||
@@ -355,7 +413,8 @@ describe('native instance provisioning', () => {
|
|||||||
it.each(['t5', 't6'] as const)('queues and verifies a %s deployment without Komodo', async game => {
|
it.each(['t5', 't6'] as const)('queues and verifies a %s deployment without Komodo', async game => {
|
||||||
const job = await queue(game); await advance();
|
const job = await queue(game); await advance();
|
||||||
const operation = await operationFor(job.deploymentId);
|
const operation = await operationFor(job.deploymentId);
|
||||||
expect(operation).toMatchObject({ kind: 'instance.create', hostId, generation: 1, state: 'pending', payload: { game, libraryId, image, port: job.port } });
|
expect(job.port).toBeNull();
|
||||||
|
expect(operation).toMatchObject({ kind: 'instance.create', hostId, generation: 1, state: 'pending', payload: { game, libraryId, image, port: 28960 } });
|
||||||
expect(operation?.payload.files).toEqual(expect.arrayContaining([expect.objectContaining({ path: 'configs/electric_cherry_map.cfg' })]));
|
expect(operation?.payload.files).toEqual(expect.arrayContaining([expect.objectContaining({ path: 'configs/electric_cherry_map.cfg' })]));
|
||||||
await acknowledge(job.deploymentId);
|
await acknowledge(job.deploymentId);
|
||||||
expect((await detail(job.deploymentId)).state).toBe('done');
|
expect((await detail(job.deploymentId)).state).toBe('done');
|
||||||
@@ -534,17 +593,22 @@ describe('native instance provisioning', () => {
|
|||||||
const { getServerScript } = await import('../../src/lib/server/fleet/inventory.js');
|
const { getServerScript } = await import('../../src/lib/server/fleet/inventory.js');
|
||||||
expect(getServerScript(job.name, 'hud.gsc')?.sha256).toBe(createHash('sha256').update(after).digest('hex'));
|
expect(getServerScript(job.name, 'hud.gsc')?.sha256).toBe(createHash('sha256').update(after).digest('hex'));
|
||||||
});
|
});
|
||||||
it('atomically launches mixed-game collections with per-host port reservations', async () => {
|
it('atomically queues mixed-game collections and reserves per-host ports when dispatched', async () => {
|
||||||
const t5 = await prepareLibrary('t5'); const t6 = await prepareLibrary('t6');
|
const t5 = await prepareLibrary('t5'); const t6 = await prepareLibrary('t6');
|
||||||
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
|
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
|
||||||
const blueprint = saveBlueprint({ name: 'Both games', entries: [{ game: 't5', mapSlug: 'theater', count: 2 }, { game: 't6', mapSlug: 'town', count: 1 }] });
|
const blueprint = saveBlueprint({ name: 'Both games', entries: [{ game: 't5', mapSlug: 'theater', count: 2 }, { game: 't6', mapSlug: 'town', count: 1 }] });
|
||||||
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
|
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
|
||||||
const preview = previewCollection(blueprint.id, 'fill', { hostId, libraryIds: { t5: t5.id, t6: t6.id }, image });
|
const preview = previewCollection(blueprint.id, 'fill', { hostId, libraryIds: { t5: t5.id, t6: t6.id }, image });
|
||||||
expect(preview.ready).toBe(true);
|
expect(preview.ready).toBe(true);
|
||||||
expect(preview.servers.map(server => server.port)).toEqual([28960, 28961, 28962]);
|
expect(preview.servers.map(server => server.port)).toEqual([null, null, null]);
|
||||||
const queued = queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name])) });
|
const queued = queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name])) });
|
||||||
expect(queueCollection({ previewId: preview.id }).collectionId).toBe(queued.collectionId);
|
expect(queueCollection({ previewId: preview.id }).collectionId).toBe(queued.collectionId);
|
||||||
const { db } = await import('../../src/lib/server/db/index.js');
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]);
|
||||||
|
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
|
||||||
|
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
|
||||||
|
saveFleetSettings({ ...fleetSettings(), createConcurrency: 4 });
|
||||||
|
await advance();
|
||||||
const instances = db().prepare('SELECT host_id,library_id,port FROM native_instances ORDER BY port').all();
|
const instances = db().prepare('SELECT host_id,library_id,port FROM native_instances ORDER BY port').all();
|
||||||
expect(instances).toEqual([{ host_id: hostId, library_id: t5.id, port: 28960 }, { host_id: hostId, library_id: t5.id, port: 28961 }, { host_id: hostId, library_id: t6.id, port: 28962 }]);
|
expect(instances).toEqual([{ host_id: hostId, library_id: t5.id, port: 28960 }, { host_id: hostId, library_id: t5.id, port: 28961 }, { host_id: hostId, library_id: t6.id, port: 28962 }]);
|
||||||
const { createHost, createEnrollment, enrollHost, hostPortAvailable } = await import('../../src/lib/server/hosts/index.js');
|
const { createHost, createEnrollment, enrollHost, hostPortAvailable } = await import('../../src/lib/server/hosts/index.js');
|
||||||
@@ -554,13 +618,15 @@ describe('native instance provisioning', () => {
|
|||||||
const { queueCreateServer } = await import('../../src/lib/server/provisioning/instances.js');
|
const { queueCreateServer } = await import('../../src/lib/server/provisioning/instances.js');
|
||||||
const second = queueCreateServer({ game: 't5', gameMode: 'zombie', mapSlug: 'moon', hostId: secondHost.id, libraryId: t5.id, image, plutoKey: 'second-host-key' });
|
const second = queueCreateServer({ game: 't5', gameMode: 'zombie', mapSlug: 'moon', hostId: secondHost.id, libraryId: t5.id, image, plutoKey: 'second-host-key' });
|
||||||
expect(second.ok).toBe(true); if (!second.ok) throw new Error(second.message);
|
expect(second.ok).toBe(true); if (!second.ok) throw new Error(second.message);
|
||||||
expect(second.port).toBe(28960);
|
expect(second.port).toBeNull();
|
||||||
|
await advance();
|
||||||
|
expect((await operationFor(second.deploymentId))?.payload.port).toBe(28960);
|
||||||
expect(hostPortAvailable(hostId, 28960)).toBe(false);
|
expect(hostPortAvailable(hostId, 28960)).toBe(false);
|
||||||
expect(hostPortAvailable(secondHost.id, 28960)).toBe(false);
|
expect(hostPortAvailable(secondHost.id, 28960)).toBe(false);
|
||||||
expect(db().prepare('SELECT COUNT(*) AS count FROM ec_host_ports').get()).toEqual({ count: 8 });
|
expect(db().prepare('SELECT COUNT(*) AS count FROM ec_host_ports').get()).toEqual({ count: 8 });
|
||||||
});
|
});
|
||||||
it('reads remote instance files through the agent including missing files and epoch timestamps', async () => {
|
it('reads remote instance files through the agent including missing files and epoch timestamps', async () => {
|
||||||
const job = await queue();
|
const job = await queue(); await advance();
|
||||||
const backend = await import('../../src/lib/server/backends/native.js');
|
const backend = await import('../../src/lib/server/backends/native.js');
|
||||||
const remoteRead = vi.mocked(backend.nativeRequest);
|
const remoteRead = vi.mocked(backend.nativeRequest);
|
||||||
remoteRead.mockImplementation(async (_instance, kind, payload) => {
|
remoteRead.mockImplementation(async (_instance, kind, payload) => {
|
||||||
@@ -622,17 +688,17 @@ describe('native instance provisioning', () => {
|
|||||||
const { getServerScript } = await import('../../src/lib/server/fleet/inventory.js');
|
const { getServerScript } = await import('../../src/lib/server/fleet/inventory.js');
|
||||||
expect(getServerScript(job.name, 'hud.gsc')).toBeUndefined();
|
expect(getServerScript(job.name, 'hud.gsc')).toBeUndefined();
|
||||||
});
|
});
|
||||||
it('cancels a queued native launch locally and releases both ports without submitting a create', async () => {
|
it('cancels an unplaced native launch locally without reserving ports or submitting a create', async () => {
|
||||||
const job = await queue();
|
const job = await queue();
|
||||||
const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
|
const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
|
||||||
const instance = nativeInstance(job.name)!;
|
expect(nativeInstance(job.name)).toBeNull();
|
||||||
const { destroyServer } = await import('../../src/lib/server/provisioning/instances.js');
|
const { destroyServer } = await import('../../src/lib/server/provisioning/instances.js');
|
||||||
const cancelled = destroyServer(job.name); expect(cancelled.ok).toBe(true);
|
const cancelled = destroyServer(job.name); expect(cancelled.ok).toBe(true);
|
||||||
await advance();
|
await advance();
|
||||||
const { db } = await import('../../src/lib/server/db/index.js');
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
expect((await detail(job.deploymentId)).state).toBe('cancelled');
|
expect((await detail(job.deploymentId)).state).toBe('cancelled');
|
||||||
expect(nativeInstance(job.name)).toBeNull();
|
expect(nativeInstance(job.name)).toBeNull();
|
||||||
expect(db().prepare('SELECT * FROM ec_host_ports WHERE instance_id=?').all(instance.id)).toEqual([]);
|
expect(db().prepare('SELECT * FROM ec_host_ports').all()).toEqual([]);
|
||||||
expect(db().prepare("SELECT COUNT(*) AS count FROM ec_host_operations WHERE kind LIKE 'instance.%'").get()).toEqual({ count: 0 });
|
expect(db().prepare("SELECT COUNT(*) AS count FROM ec_host_operations WHERE kind LIKE 'instance.%'").get()).toEqual({ count: 0 });
|
||||||
const { getLibraryVersion } = await import('../../src/lib/server/game-files/index.js');
|
const { getLibraryVersion } = await import('../../src/lib/server/game-files/index.js');
|
||||||
expect(getLibraryVersion(libraryId)?.state).toBe('ready');
|
expect(getLibraryVersion(libraryId)?.state).toBe('ready');
|
||||||
@@ -661,7 +727,8 @@ describe('automatic capacity admission and placement', () => {
|
|||||||
const { recipeForDeployment } = await import('../../src/lib/server/fleet/launch-recipes.js');
|
const { recipeForDeployment } = await import('../../src/lib/server/fleet/launch-recipes.js');
|
||||||
const result = queueCreateServer(input);
|
const result = queueCreateServer(input);
|
||||||
expect(result.ok).toBe(true); if (!result.ok) return;
|
expect(result.ok).toBe(true); if (!result.ok) return;
|
||||||
expect(nativeInstance(result.name)?.host_id).toBe(hostId); // Fast first fit.
|
expect(nativeInstance(result.name)).toBeNull();
|
||||||
|
expect(result.port).toBeNull();
|
||||||
await advance();
|
await advance();
|
||||||
expect((await operationFor(result.deploymentId))?.hostId).toBe(other.id);
|
expect((await operationFor(result.deploymentId))?.hostId).toBe(other.id);
|
||||||
expect(nativeInstance(result.name)).toMatchObject({ host_id: other.id, public_address: 'second.example' });
|
expect(nativeInstance(result.name)).toMatchObject({ host_id: other.id, public_address: 'second.example' });
|
||||||
@@ -689,13 +756,18 @@ describe('automatic capacity admission and placement', () => {
|
|||||||
it.each([
|
it.each([
|
||||||
{ cpuUsagePercent: undefined }, { memoryAvailable: undefined }, { cpuCount: 0 },
|
{ cpuUsagePercent: undefined }, { memoryAvailable: undefined }, { cpuCount: 0 },
|
||||||
{ cpuUsagePercent: 99 }, { memoryAvailable: 1024 ** 3 }
|
{ cpuUsagePercent: 99 }, { memoryAvailable: 1024 ** 3 }
|
||||||
])('rejects admission with unavailable resources: %j', async overrides => {
|
])('accepts an unplaced job and waits for unavailable resources: %j', async overrides => {
|
||||||
await prepareLibrary();
|
await prepareLibrary();
|
||||||
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
|
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
|
||||||
heartbeat(hostId, { ...inventory(), ...overrides });
|
heartbeat(hostId, { ...inventory(), ...overrides });
|
||||||
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
|
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
|
||||||
const { db } = await import('../../src/lib/server/db/index.js');
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
expect(queueNativeCreation(input)).toMatchObject({ ok: false, kind: 'conflict', message: expect.stringContaining('capacity') });
|
const queued = queueNativeCreation(input);
|
||||||
|
expect(queued).toMatchObject({ ok: true, port: null });
|
||||||
|
if (!queued.ok) throw new Error(queued.message);
|
||||||
|
await advance();
|
||||||
|
expect(await operationFor(queued.deploymentId)).toBeNull();
|
||||||
|
expect((await detail(queued.deploymentId)).stage).toContain('capacity');
|
||||||
expect(db().prepare('SELECT * FROM native_instances').all()).toEqual([]);
|
expect(db().prepare('SELECT * FROM native_instances').all()).toEqual([]);
|
||||||
expect(db().prepare('SELECT * FROM ec_host_ports').all()).toEqual([]);
|
expect(db().prepare('SELECT * FROM ec_host_ports').all()).toEqual([]);
|
||||||
});
|
});
|
||||||
@@ -709,22 +781,28 @@ describe('automatic capacity admission and placement', () => {
|
|||||||
const { db } = await import('../../src/lib/server/db/index.js');
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
db().prepare("UPDATE ec_hosts SET last_heartbeat='2000-01-01' WHERE id=?").run(hostId);
|
db().prepare("UPDATE ec_hosts SET last_heartbeat='2000-01-01' WHERE id=?").run(hostId);
|
||||||
const result = queueNativeCreation({ ...input, previewId: preview.id });
|
const result = queueNativeCreation({ ...input, previewId: preview.id });
|
||||||
expect(result).toMatchObject({ ok: true, name: preview.name, port: 30000 });
|
expect(result).toMatchObject({ ok: true, name: preview.name, port: null });
|
||||||
expect(queueNativeCreation({ ...input, previewId: preview.id })).toEqual(result);
|
expect(queueNativeCreation({ ...input, previewId: preview.id })).toEqual(result);
|
||||||
|
if (!result.ok) throw new Error(result.message);
|
||||||
|
await advance();
|
||||||
|
expect((await operationFor(result.deploymentId))?.payload.port).toBe(30000);
|
||||||
});
|
});
|
||||||
it('rejects an entire collection when capacity shrinks after its preview', async () => {
|
it('queues the whole collection and waits when capacity shrinks after its preview', async () => {
|
||||||
await prepareLibrary();
|
await prepareLibrary();
|
||||||
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
|
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
|
||||||
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
|
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
|
||||||
const blueprint = saveBlueprint({ name: 'Two lobbies', entries: [{ game: 't5', mapSlug: 'theater', count: 2 }] });
|
const blueprint = saveBlueprint({ name: 'Two lobbies', entries: [{ game: 't5', mapSlug: 'theater', count: 2 }] });
|
||||||
const preview = previewCollection(blueprint.id);
|
const preview = previewCollection(blueprint.id);
|
||||||
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
|
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
|
||||||
heartbeat(hostId, { ...inventory(), cpuCount: 1, cpuUsagePercent: 0 });
|
heartbeat(hostId, { ...inventory(), cpuUsagePercent: 100 });
|
||||||
expect(() => queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key'])) })).toThrow('capacity');
|
const result = queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key'])) });
|
||||||
|
expect(result).toMatchObject({ ok: true, collectionId: expect.any(String) });
|
||||||
|
await advance();
|
||||||
const { db } = await import('../../src/lib/server/db/index.js');
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
expect(db().prepare('SELECT * FROM native_instances').all()).toEqual([]);
|
expect(db().prepare('SELECT * FROM native_instances').all()).toEqual([]);
|
||||||
expect(db().prepare('SELECT * FROM ec_host_ports').all()).toEqual([]);
|
expect(db().prepare('SELECT * FROM ec_host_ports').all()).toEqual([]);
|
||||||
expect(db().prepare('SELECT * FROM server_launch_collections').all()).toEqual([]);
|
expect(db().prepare('SELECT * FROM server_launch_collections').all()).toHaveLength(1);
|
||||||
|
expect(db().prepare('SELECT * FROM native_creation_intents').all()).toHaveLength(2);
|
||||||
});
|
});
|
||||||
it.each([
|
it.each([
|
||||||
{ resource: 'CPU', measurements: { cpuCount: 4, cpuUsagePercent: 82.5 }, over: { cpuUsagePercent: 82.51 } },
|
{ resource: 'CPU', measurements: { cpuCount: 4, cpuUsagePercent: 82.5 }, over: { cpuUsagePercent: 82.51 } },
|
||||||
@@ -742,15 +820,19 @@ describe('automatic capacity admission and placement', () => {
|
|||||||
expect(() => choosePlacement(options)).toThrow('capacity');
|
expect(() => choosePlacement(options)).toThrow('capacity');
|
||||||
expect(() => choosePlacement({ ...options, thorough: true })).toThrow('capacity');
|
expect(() => choosePlacement({ ...options, thorough: true })).toThrow('capacity');
|
||||||
});
|
});
|
||||||
it('uses calibrated footprints to admit more instances than the cold-start allowance', async () => {
|
it('dispatches a waiting instance after calibration lowers the cold-start allowance', async () => {
|
||||||
const first = await queue();
|
const first = await queue();
|
||||||
|
await advance(); await acknowledge(first.deploymentId);
|
||||||
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
|
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
|
||||||
const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
|
const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
|
||||||
const { recordInstanceResources, capacityCalibration } = await import('../../src/lib/server/hosts/calibration.js');
|
const { recordInstanceResources, capacityCalibration } = await import('../../src/lib/server/hosts/calibration.js');
|
||||||
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
|
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
|
||||||
const measured = { ...inventory(), cpuCount: 1, cpuUsagePercent: 0 };
|
const measured = { ...inventory(), cpuCount: 1, cpuUsagePercent: 50 };
|
||||||
heartbeat(hostId, measured);
|
heartbeat(hostId, measured);
|
||||||
expect(queueNativeCreation(input)).toMatchObject({ ok: false, kind: 'conflict' });
|
const next = queueNativeCreation(input);
|
||||||
|
expect(next.ok).toBe(true); if (!next.ok) throw new Error(next.message);
|
||||||
|
await advance();
|
||||||
|
expect(await operationFor(next.deploymentId)).toBeNull();
|
||||||
const instance = nativeInstance(first.name)!;
|
const instance = nativeInstance(first.name)!;
|
||||||
for (let minute = 0; minute < 25; minute++) {
|
for (let minute = 0; minute < 25; minute++) {
|
||||||
recordInstanceResources(hostId, { ...measured, instanceResources: [{ instanceId: instance.id, generation: 1,
|
recordInstanceResources(hostId, { ...measured, instanceResources: [{ instanceId: instance.id, generation: 1,
|
||||||
@@ -758,7 +840,8 @@ describe('automatic capacity admission and placement', () => {
|
|||||||
new Date(Date.now() - (25 - minute) * 60_000).toISOString());
|
new Date(Date.now() - (25 - minute) * 60_000).toISOString());
|
||||||
}
|
}
|
||||||
expect(capacityCalibration(hostId).startup.source).toBe('host');
|
expect(capacityCalibration(hostId).startup.source).toBe('host');
|
||||||
expect(queueNativeCreation(input)).toMatchObject({ ok: true });
|
await advance();
|
||||||
|
expect(await operationFor(next.deploymentId)).not.toBeNull();
|
||||||
});
|
});
|
||||||
it('queues a collection at running demand and releases startup reservations as live usage arrives', async () => {
|
it('queues a collection at running demand and releases startup reservations as live usage arrives', async () => {
|
||||||
await prepareLibrary();
|
await prepareLibrary();
|
||||||
@@ -791,7 +874,7 @@ describe('automatic capacity admission and placement', () => {
|
|||||||
await advance();
|
await advance();
|
||||||
expect(creates()).toHaveLength(2); // Its actual usage replaces the peak even before health verification.
|
expect(creates()).toHaveLength(2); // Its actual usage replaces the peak even before health verification.
|
||||||
});
|
});
|
||||||
it('charges queued reservations and releases them on cancellation', async () => {
|
it('reserves queued lobby identities without host resources and releases them on cancellation', async () => {
|
||||||
await prepareLibrary();
|
await prepareLibrary();
|
||||||
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
|
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
|
||||||
heartbeat(hostId, { ...inventory(), cpuCount: 1, cpuUsagePercent: 0 });
|
heartbeat(hostId, { ...inventory(), cpuCount: 1, cpuUsagePercent: 0 });
|
||||||
@@ -799,9 +882,13 @@ describe('automatic capacity admission and placement', () => {
|
|||||||
const { cancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
|
const { cancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
|
||||||
const first = queueNativeCreation(input);
|
const first = queueNativeCreation(input);
|
||||||
expect(first.ok).toBe(true); if (!first.ok) return;
|
expect(first.ok).toBe(true); if (!first.ok) return;
|
||||||
expect(queueNativeCreation(input)).toMatchObject({ ok: false, kind: 'conflict' });
|
const second = queueNativeCreation(input);
|
||||||
|
expect(second).toMatchObject({ ok: true, name: 't5_zombie_theater_2', port: null });
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]);
|
||||||
|
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
|
||||||
cancelCreation(first.deploymentId);
|
cancelCreation(first.deploymentId);
|
||||||
expect(queueNativeCreation(input)).toMatchObject({ ok: true });
|
expect(queueNativeCreation(input)).toMatchObject({ ok: true, name: first.name, port: null });
|
||||||
});
|
});
|
||||||
it.each(['memory', 'cpu'] as const)('dispatches immediately after %s recovers instead of waiting for five-minute history', async resource => {
|
it.each(['memory', 'cpu'] as const)('dispatches immediately after %s recovers instead of waiting for five-minute history', async resource => {
|
||||||
await prepareLibrary();
|
await prepareLibrary();
|
||||||
@@ -846,7 +933,10 @@ describe('automatic capacity admission and placement', () => {
|
|||||||
// even though the next host heartbeat has not observed it yet.
|
// even though the next host heartbeat has not observed it yet.
|
||||||
await advance();
|
await advance();
|
||||||
expect(db().prepare("SELECT * FROM ec_host_operations WHERE kind='instance.create'").all()).toHaveLength(1);
|
expect(db().prepare("SELECT * FROM ec_host_operations WHERE kind='instance.create'").all()).toHaveLength(1);
|
||||||
expect(queueNativeCreation(input)).toMatchObject({ ok: false, kind: 'conflict' });
|
const later = queueNativeCreation(input);
|
||||||
|
expect(later.ok).toBe(true); if (!later.ok) throw new Error(later.message);
|
||||||
|
await advance();
|
||||||
|
expect(await operationFor(later.deploymentId)).toBeNull();
|
||||||
});
|
});
|
||||||
it('does not charge a starting container twice when its usage is already in the heartbeat', async () => {
|
it('does not charge a starting container twice when its usage is already in the heartbeat', async () => {
|
||||||
await prepareLibrary();
|
await prepareLibrary();
|
||||||
@@ -873,6 +963,8 @@ describe('automatic capacity admission and placement', () => {
|
|||||||
});
|
});
|
||||||
it.each(['missing', 'old-generation'] as const)('keeps full reservations for a submitted launch with %s measurements', async evidence => {
|
it.each(['missing', 'old-generation'] as const)('keeps full reservations for a submitted launch with %s measurements', async evidence => {
|
||||||
await prepareLibrary();
|
await prepareLibrary();
|
||||||
|
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
|
||||||
|
saveFleetSettings({ ...fleetSettings(), createConcurrency: 2 });
|
||||||
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
|
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
|
||||||
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
|
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
|
||||||
const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
|
const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
|
||||||
@@ -885,8 +977,11 @@ describe('automatic capacity admission and placement', () => {
|
|||||||
heartbeat(hostId, { ...inventory(), memoryTotal: 3 * GiB, memoryAvailable: 2 * GiB,
|
heartbeat(hostId, { ...inventory(), memoryTotal: 3 * GiB, memoryAvailable: 2 * GiB,
|
||||||
instanceResources: evidence === 'missing' ? [] : [{ instanceId: instance.id, generation: instance.generation + 1,
|
instanceResources: evidence === 'missing' ? [] : [{ instanceId: instance.id, generation: instance.generation + 1,
|
||||||
uptimeSeconds: 60, cpuCores: 0.1, memoryBytes: Math.floor(0.8 * GiB) }] });
|
uptimeSeconds: 60, cpuCores: 0.1, memoryBytes: Math.floor(0.8 * GiB) }] });
|
||||||
expect(queueNativeCreation(input)).toMatchObject({ ok: false, kind: 'conflict',
|
const next = queueNativeCreation(input);
|
||||||
message: expect.stringContaining('3.00 GiB projected, 2.53 GiB usable') });
|
expect(next.ok).toBe(true); if (!next.ok) throw new Error(next.message);
|
||||||
|
await advance();
|
||||||
|
expect(await operationFor(next.deploymentId)).toBeNull();
|
||||||
|
expect((await detail(next.deploymentId)).stage).toContain('3.00 GiB projected, 2.53 GiB usable');
|
||||||
});
|
});
|
||||||
it('waits when capacity disappears after admission without sending a create command', async () => {
|
it('waits when capacity disappears after admission without sending a create command', async () => {
|
||||||
await prepareLibrary();
|
await prepareLibrary();
|
||||||
@@ -931,11 +1026,16 @@ describe('automatic capacity admission and placement', () => {
|
|||||||
const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
|
const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
|
||||||
const result = queueNativeCreation(input);
|
const result = queueNativeCreation(input);
|
||||||
expect(result.ok).toBe(true); if (!result.ok) return;
|
expect(result.ok).toBe(true); if (!result.ok) return;
|
||||||
|
expect(nativeInstance(result.name)).toBeNull();
|
||||||
|
await advance();
|
||||||
expect(nativeInstance(result.name)?.host_id).toBe(other.id);
|
expect(nativeInstance(result.name)?.host_id).toBe(other.id);
|
||||||
updateHost(other.id, { mode: 'draining' });
|
updateHost(other.id, { mode: 'draining' });
|
||||||
expect(queueNativeCreation(input)).toMatchObject({ ok: false, kind: 'conflict' });
|
const next = queueNativeCreation(input);
|
||||||
|
expect(next.ok).toBe(true); if (!next.ok) throw new Error(next.message);
|
||||||
|
await advance();
|
||||||
|
expect(await operationFor(next.deploymentId)).toBeNull();
|
||||||
});
|
});
|
||||||
it('spreads collection admission across hosts, allowing the same port on different hosts', async () => {
|
it('spreads collection dispatch across hosts, allowing the same port on different hosts', async () => {
|
||||||
await prepareLibrary(); await secondHost({ cpuCount: 1, cpuUsagePercent: 0 });
|
await prepareLibrary(); await secondHost({ cpuCount: 1, cpuUsagePercent: 0 });
|
||||||
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
|
const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
|
||||||
heartbeat(hostId, { ...inventory(), cpuCount: 1, cpuUsagePercent: 0 });
|
heartbeat(hostId, { ...inventory(), cpuCount: 1, cpuUsagePercent: 0 });
|
||||||
@@ -944,10 +1044,17 @@ describe('automatic capacity admission and placement', () => {
|
|||||||
const blueprint = saveBlueprint({ name: 'Two lobbies', entries: [{ game: 't5', mapSlug: 'theater', count: 2 }] });
|
const blueprint = saveBlueprint({ name: 'Two lobbies', entries: [{ game: 't5', mapSlug: 'theater', count: 2 }] });
|
||||||
const preview = previewCollection(blueprint.id);
|
const preview = previewCollection(blueprint.id);
|
||||||
expect(preview.ready).toBe(true);
|
expect(preview.ready).toBe(true);
|
||||||
expect(preview.servers.map(server => server.port)).toEqual([28960, 28960]);
|
expect(preview.servers.map(server => server.port)).toEqual([null, null]);
|
||||||
expect(new Set(preview.servers.map(server => server.native?.hostId)).size).toBe(2);
|
|
||||||
expect(queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key'])) }).ok).toBe(true);
|
expect(queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key'])) }).ok).toBe(true);
|
||||||
const { db } = await import('../../src/lib/server/db/index.js');
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]);
|
||||||
|
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
|
||||||
|
saveFleetSettings({ ...fleetSettings(), createConcurrency: 2 });
|
||||||
|
await advance();
|
||||||
|
// Concurrent ranking can select the same host before the first
|
||||||
|
// reservation commits. The held job retries against the new usage.
|
||||||
|
await advance();
|
||||||
expect(db().prepare('SELECT DISTINCT host_id FROM native_instances').all()).toHaveLength(2);
|
expect(db().prepare('SELECT DISTINCT host_id FROM native_instances').all()).toHaveLength(2);
|
||||||
|
expect(db().prepare('SELECT port FROM native_instances').all()).toEqual([{ port: 28960 }, { port: 28960 }]);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
Reference in New Issue
Block a user