Add cancellation for collection launches
This commit is contained in:
@@ -2,7 +2,7 @@
|
||||
|
||||
Open **Fleet → Collections** to manage reusable server lineups. Use
|
||||
**Active launches** to follow progress and **Launch history** to review completed
|
||||
launches and failures. Both views refresh automatically. A
|
||||
launches, cancellations, and failures. Both views refresh automatically. A
|
||||
collection describes a whole lineup: for example, two Kino lobbies, one Moon lobby,
|
||||
and two Town lobbies. Counts are desired fleet totals: existing and queued
|
||||
servers count toward each target, and only missing servers are created.
|
||||
@@ -153,13 +153,37 @@ running. A confirmed creation failure is recorded for review, allowing later
|
||||
queued servers to proceed on subsequent passes without undoing successful launches.
|
||||
|
||||
The collection launch page at `/collections/launches/:id` separates ready,
|
||||
launching, queued, and failed servers. Its breadcrumb returns to Collections. It
|
||||
launching, queued, failed, and cancelled servers. Its breadcrumb returns to Collections. It
|
||||
highlights the current startup, shows each waiting server's queue position, and
|
||||
links to earlier launches when another collection occupies the available capacity.
|
||||
Every server links to its Provisioning operation timeline and recovery options,
|
||||
and each child operation links back to its collection launch. Individual operations
|
||||
remain visible in Provisioning activity and history.
|
||||
|
||||
### Cancel a collection launch
|
||||
|
||||
Operators and administrators can choose **Cancel collection launch** on the
|
||||
collection progress page while eligible work remains. Review and confirm the
|
||||
cleanup before proceeding. Cancellation stops unfinished creations, including
|
||||
eligible failed or interrupted launches, and removes their containers and instance
|
||||
files. This can disconnect players who joined before startup verification finished,
|
||||
even when player status is unknown. Shared game files are kept. Keys and ports
|
||||
remain reserved until removal is confirmed.
|
||||
|
||||
Completed launches, including those that finished with a warning, and retained
|
||||
servers are kept. In exact mode, safe pending removals belonging to this launch
|
||||
are cancelled. Removals that have already started or are still needed by another
|
||||
collection may finish; cancellation does not restore servers already removed.
|
||||
|
||||
The progress page shows **Cancellation requested** while work or cleanup remains,
|
||||
then **Collection launch cancelled**, or **Cancellation needs attention** if
|
||||
operations or cleanup failed. Cancelled operations are counted separately
|
||||
from failures. Each cancelled creation links to its cleanup operation for progress
|
||||
and recovery. A launch stays in **Active launches** until its work and cleanup
|
||||
have settled; failed cleanup remains marked as needing attention in launch history.
|
||||
Cancellation does not delete the saved collection, so it can be previewed again
|
||||
against the remaining fleet.
|
||||
|
||||
## Create a single server
|
||||
|
||||
1. Open **Create server** from the fleet or **Single server** from Provisioning.
|
||||
@@ -296,11 +320,17 @@ there while preserving query parameters.
|
||||
A satisfied preview returns `{ ok: true, collectionId: null }` with HTTP 200,
|
||||
without creating a collection, saving keys, or kicking the worker.
|
||||
- `GET /api/deployment-collections`: list collection progress summaries with
|
||||
separate `queued`, `provisioning`, and `verifying` counts.
|
||||
separate `queued`, `provisioning`, `verifying`, and `cancelled` counts,
|
||||
`cleanupActive` and `cleanupFailed` counts, and `cancelled_at` when cancellation
|
||||
was requested. Cancelled operations are excluded from `failed`.
|
||||
- `GET /api/deployment-collections/:id`: collection mode, retained server IDs,
|
||||
per-operation progress with `action: create | destroy`,
|
||||
each waiting server's global `queuePosition`, and the shared queue's concurrency,
|
||||
active startups, and queued count.
|
||||
active startups, and queued count. Includes `canCancel` and each creation's
|
||||
`cancellation` cleanup ID, state, stage, and error when applicable.
|
||||
- `POST /api/deployment-collections/:id/cancel`: request cancellation and cleanup
|
||||
of eligible unfinished work. Returns `{ ok: true }`, HTTP 404 for an unknown
|
||||
launch, or HTTP 409 when the launch cannot be cancelled.
|
||||
- `POST /api/deployments/preview`: preview `{ game, mapSlug }` or `{ deploymentId }`
|
||||
for a new single server using the latest promoted setup. All collections, including one-server
|
||||
targets, must use the collection endpoint so existing servers count correctly.
|
||||
|
||||
@@ -68,9 +68,13 @@ export interface CollectionSummary {
|
||||
id: string;
|
||||
name: string;
|
||||
created_at: string;
|
||||
cancelled_at: string | null;
|
||||
total: number;
|
||||
done: number;
|
||||
failed: number;
|
||||
cancelled: number;
|
||||
cleanupActive: number;
|
||||
cleanupFailed: number;
|
||||
active: number;
|
||||
queued: number;
|
||||
provisioning: number;
|
||||
@@ -80,11 +84,13 @@ export interface CollectionSummary {
|
||||
|
||||
export interface CollectionDetail {
|
||||
collection: CollectionSummary;
|
||||
canCancel: boolean;
|
||||
retainedServers: string[];
|
||||
retainedServerDeletedAt?: Record<string, string>;
|
||||
queue: { concurrency: number; active: { id: string; server: string; state: string; stage: string }[]; queued: number };
|
||||
servers: { id: string; server: string; action: 'create' | 'destroy'; port: number | null; map_slug: string | null; state: string; stage: string;
|
||||
started_at: string; finished_at: string | null; error: string | null; server_exists: number; server_deleted_at?: string | null; queuePosition: number | null }[];
|
||||
started_at: string; finished_at: string | null; error: string | null; server_exists: number; server_deleted_at?: string | null; queuePosition: number | null;
|
||||
cancellation: { cleanupId: string; state: string; stage: string; error: string | null } | null }[];
|
||||
}
|
||||
|
||||
export interface LaunchPreview {
|
||||
|
||||
@@ -368,6 +368,7 @@ export function db(): Database.Database {
|
||||
// Older databases gain rolled_back_by above, after schema.sql runs.
|
||||
handle.exec('CREATE INDEX IF NOT EXISTS idx_deployments_rolled_back_by ON deployments(rolled_back_by)');
|
||||
migrateTable(handle, 'server_launch_collections', [
|
||||
{ name: 'cancelled_at', ddl: 'ALTER TABLE server_launch_collections ADD COLUMN cancelled_at TEXT' },
|
||||
{ name: 'mode', ddl: "ALTER TABLE server_launch_collections ADD COLUMN mode TEXT NOT NULL DEFAULT 'fill'" },
|
||||
{ name: 'retained_servers', ddl: "ALTER TABLE server_launch_collections ADD COLUMN retained_servers TEXT NOT NULL DEFAULT '[]'" }
|
||||
]);
|
||||
|
||||
@@ -581,6 +581,7 @@ CREATE TABLE IF NOT EXISTS server_launch_collections (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
cancelled_at TEXT,
|
||||
mode TEXT NOT NULL DEFAULT 'fill',
|
||||
retained_servers TEXT NOT NULL DEFAULT '[]',
|
||||
retained_server_deletions TEXT NOT NULL DEFAULT '{}'
|
||||
|
||||
@@ -14,6 +14,8 @@ import { nativeLaunchSelection, queueNativeCreationIntent } from '../provisionin
|
||||
import { nativeDeployment, nativeInstance } from '../backends/native.js';
|
||||
import { launchAvailability } from '../hosts/availability.js';
|
||||
import { availableServerKeyCount } from './server-keys.js';
|
||||
import { canCancelCreation, cancelCreation } from '../provisioning/cancel-creation.js';
|
||||
import { canCancelRemoval, cancelRemoval } from '../provisioning/history.js';
|
||||
|
||||
function coverage(entries: BlueprintEntry[], includeHealth = true) {
|
||||
type Server = { name: string; game: string | null; map_slug: string | null; map: string | null };
|
||||
@@ -240,16 +242,61 @@ export function queueCollection(input: { previewId?: unknown; keys?: unknown }):
|
||||
const membershipSql = `SELECT collection_id, deployment_id, position FROM server_collection_members
|
||||
UNION ALL SELECT collection_id, deployment_id, position FROM server_collection_removals`;
|
||||
|
||||
const summarySql = `SELECT c.id, c.name, c.created_at, c.mode, COUNT(d.id) AS total,
|
||||
function collectionCancellationCandidates(id: string) {
|
||||
const creations = (db().prepare(`SELECT d.id FROM server_collection_members m
|
||||
JOIN server_deployments d ON d.id=m.deployment_id
|
||||
WHERE m.collection_id=?
|
||||
AND NOT EXISTS (SELECT 1 FROM native_creation_cancellations cc WHERE cc.creation_id=d.id)`)
|
||||
.all(id) as { id: string }[]).filter(job => canCancelCreation(job.id));
|
||||
// Exact launches can reuse manual removals or creation cleanup jobs. Membership
|
||||
// alone must never give a launch permission to withdraw those operations.
|
||||
const removals = (db().prepare(`SELECT d.id FROM server_collection_removals m
|
||||
JOIN server_deployments d ON d.id=m.deployment_id
|
||||
WHERE m.collection_id=? AND d.triggered_by='blueprint-exact'
|
||||
AND NOT EXISTS (SELECT 1 FROM native_provisioning n WHERE n.deployment_id=d.id AND n.operation_id IS NOT NULL)
|
||||
AND NOT EXISTS (SELECT 1 FROM native_creation_cancellations cc WHERE cc.cleanup_id=d.id)`)
|
||||
.all(id) as { id: string }[]).filter(job => canCancelRemoval(job.id));
|
||||
return { creations, removals };
|
||||
}
|
||||
|
||||
/** Withdraw the remaining launch atomically; partial instances keep durable cleanup jobs. */
|
||||
export function cancelCollection(id: string): boolean {
|
||||
return db().transaction(() => {
|
||||
const collection = db().prepare('SELECT cancelled_at FROM server_launch_collections WHERE id=?').get(id) as { cancelled_at: string | null } | undefined;
|
||||
if (!collection) return false;
|
||||
if (collection.cancelled_at) return true;
|
||||
const { creations, removals } = collectionCancellationCandidates(id);
|
||||
if (!creations.length && !removals.length) return false;
|
||||
db().prepare('UPDATE server_launch_collections SET cancelled_at=? WHERE id=?').run(new Date().toISOString(), id);
|
||||
for (const creation of creations) {
|
||||
if (!cancelCreation(creation.id, 'collection-cancel')) throw new Error('Could not cancel a collection creation. Refresh the launch and try again.');
|
||||
}
|
||||
for (const removal of removals) {
|
||||
// A shared removal remains requested until every owning launch withdraws.
|
||||
const shared = db().prepare(`SELECT 1 FROM server_collection_removals m
|
||||
JOIN server_launch_collections c ON c.id=m.collection_id
|
||||
WHERE m.deployment_id=? AND c.id!=? AND c.cancelled_at IS NULL LIMIT 1`).get(removal.id, id);
|
||||
if (!shared) cancelRemoval(removal.id);
|
||||
}
|
||||
return true;
|
||||
})();
|
||||
}
|
||||
|
||||
const summarySql = `SELECT c.id, c.name, c.created_at, c.cancelled_at, c.mode, COUNT(d.id) AS total,
|
||||
COALESCE(SUM(d.state IN ('done', 'warning')), 0) AS done,
|
||||
COALESCE(SUM(d.state IN ('failed', 'interrupted', 'cancelled')), 0) AS failed,
|
||||
COALESCE(SUM(d.state IN ('failed', 'interrupted')), 0) AS failed,
|
||||
COALESCE(SUM(d.state = 'cancelled'), 0) AS cancelled,
|
||||
COALESCE(SUM(cleanup.state IN ('queued', 'running', 'awaiting-health', 'waiting')), 0) AS cleanupActive,
|
||||
COALESCE(SUM(cleanup.state IN ('failed', 'interrupted', 'warning', 'cancelled')), 0) AS cleanupFailed,
|
||||
COALESCE(SUM(d.state IN ('queued', 'running', 'awaiting-health', 'waiting')), 0) AS active,
|
||||
COALESCE(SUM(d.state IN ('queued', 'waiting')), 0) AS queued,
|
||||
COALESCE(SUM(d.state = 'running'), 0) AS provisioning,
|
||||
COALESCE(SUM(d.state = 'awaiting-health'), 0) AS verifying
|
||||
FROM server_launch_collections c
|
||||
LEFT JOIN (${membershipSql}) m ON m.collection_id = c.id
|
||||
LEFT JOIN server_deployments d ON d.id = m.deployment_id`;
|
||||
LEFT JOIN server_deployments d ON d.id = m.deployment_id
|
||||
LEFT JOIN native_creation_cancellations cc ON cc.creation_id = d.id
|
||||
LEFT JOIN server_deployments cleanup ON cleanup.id = cc.cleanup_id`;
|
||||
|
||||
export function listCollections(): CollectionSummary[] {
|
||||
return db().prepare(summarySql + ' GROUP BY c.id ORDER BY c.created_at DESC, c.rowid DESC').all() as CollectionSummary[];
|
||||
@@ -265,6 +312,13 @@ export function getCollection(id: string): CollectionDetail | null {
|
||||
const queue = creationQueue();
|
||||
const positions = new Map(queue.queued.map((job, index) => [job.id, index + 1]));
|
||||
const stored = db().prepare('SELECT retained_servers, retained_server_deletions FROM server_launch_collections WHERE id = ?').get(id) as { retained_servers: string; retained_server_deletions: string };
|
||||
return { collection, retainedServers: JSON.parse(stored.retained_servers), retainedServerDeletedAt: JSON.parse(stored.retained_server_deletions), queue: { concurrency: fleetSettings().createConcurrency, active: queue.active, queued: queue.queued.length },
|
||||
servers: servers.map(server => ({ ...server, queuePosition: positions.get(server.id) ?? null })) };
|
||||
const cancellations = db().prepare(`SELECT cc.creation_id, cc.cleanup_id AS cleanupId, d.state, d.stage, d.error
|
||||
FROM server_collection_members m JOIN native_creation_cancellations cc ON cc.creation_id=m.deployment_id
|
||||
JOIN server_deployments d ON d.id=cc.cleanup_id WHERE m.collection_id=?`).all(id) as
|
||||
(NonNullable<CollectionDetail['servers'][number]['cancellation']> & { creation_id: string })[];
|
||||
const cleanupByCreation = new Map(cancellations.map(({ creation_id, ...cleanup }) => [creation_id, cleanup]));
|
||||
const candidates = collection.cancelled_at ? null : collectionCancellationCandidates(id);
|
||||
return { collection, canCancel: !!candidates && (candidates.creations.length > 0 || candidates.removals.length > 0),
|
||||
retainedServers: JSON.parse(stored.retained_servers), retainedServerDeletedAt: JSON.parse(stored.retained_server_deletions), queue: { concurrency: fleetSettings().createConcurrency, active: queue.active, queued: queue.queued.length },
|
||||
servers: servers.map(server => ({ ...server, queuePosition: positions.get(server.id) ?? null, cancellation: cleanupByCreation.get(server.id) ?? null })) };
|
||||
}
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
import { json } from '@sveltejs/kit';
|
||||
import { cancelCollection } from '$lib/server/fleet/collections.js';
|
||||
import { db } from '$lib/server/db/index.js';
|
||||
import { kickProvisioning } from '$lib/server/pipeline/worker.js';
|
||||
|
||||
export function POST({ params }: { params: { id: string } }) {
|
||||
if (!db().prepare('SELECT 1 FROM server_launch_collections WHERE id=?').get(params.id)) {
|
||||
return json({ error: 'Collection not found.' }, { status: 404 });
|
||||
}
|
||||
if (!cancelCollection(params.id)) {
|
||||
return json({ error: 'This collection has no work that can still be cancelled. Refresh to see its current state.' }, { status: 409 });
|
||||
}
|
||||
void kickProvisioning();
|
||||
return json({ ok: true });
|
||||
}
|
||||
@@ -31,14 +31,27 @@
|
||||
let limit = $state(25);
|
||||
let launchLimit = $state(10);
|
||||
const canOperate = $derived($page.data.user?.role === 'admin' || $page.data.user?.role === 'operator');
|
||||
const activeLaunches = $derived(launches.filter(launch => launch.active > 0));
|
||||
const launchHistory = $derived(launches.filter(launch => launch.active === 0));
|
||||
const inProgress = (launch: CollectionSummary) => launch.active > 0 || launch.cleanupActive > 0;
|
||||
const activeLaunches = $derived(launches.filter(inProgress));
|
||||
const launchHistory = $derived(launches.filter(launch => !inProgress(launch)));
|
||||
const visibleLaunches = $derived(tab === 'active' ? activeLaunches : launchHistory);
|
||||
const tabs = $derived([
|
||||
{ id: 'collections', label: 'Your collections' },
|
||||
{ id: 'active', label: 'Active launches', badge: launchesLoading ? undefined : activeLaunches.length },
|
||||
{ id: 'history', label: 'Launch history' }
|
||||
]);
|
||||
function launchLabel(collection: CollectionSummary) {
|
||||
if (collection.cancelled_at) return inProgress(collection) ? 'Cancellation requested' : collection.failed || collection.cleanupFailed ? 'Cancelled · needs attention' : 'Cancelled';
|
||||
if (collection.cleanupActive) return 'Cleaning up';
|
||||
if (collection.active) return collection.provisioning + collection.verifying ? collection.mode === 'exact' ? 'Applying collection' : 'Launching' : 'Queued';
|
||||
if (collection.failed || collection.cleanupFailed) return 'Needs attention';
|
||||
return collection.cancelled ? 'Finished with cancellations' : 'Completed';
|
||||
}
|
||||
function launchVariant(collection: CollectionSummary) {
|
||||
if (collection.cleanupFailed || (!inProgress(collection) && collection.failed)) return 'danger';
|
||||
if (inProgress(collection)) return 'info';
|
||||
return collection.cancelled_at || collection.cancelled ? 'neutral' : 'online';
|
||||
}
|
||||
$effect(() => {
|
||||
const params = $page.url.searchParams;
|
||||
tab = ['active', 'history'].includes(params.get('view') ?? '') ? params.get('view')! : 'collections';
|
||||
@@ -136,14 +149,14 @@
|
||||
{:else if visibleLaunches.length}
|
||||
<div class="launches">{#each visibleLaunches.slice(0,launchLimit) as collection (collection.id)}
|
||||
<a class="collection-row" href={contextHref(`/collections/launches/${collection.id}`, $page.url)}>
|
||||
<div class="row-head"><strong>{collection.name}</strong><Badge variant={collection.active ? 'info' : collection.failed ? 'danger' : 'online'}>{collection.active ? collection.provisioning + collection.verifying ? collection.mode === 'exact' ? 'Applying collection' : 'Launching' : 'Queued' : collection.failed ? 'Needs attention' : 'Completed'}</Badge></div>
|
||||
<div class="collection-counts"><span><strong>{collection.done} / {collection.total}</strong> {collection.mode === 'exact' ? 'completed' : 'ready'}</span>{#if collection.provisioning + collection.verifying}<span>{collection.provisioning + collection.verifying} {collection.mode === 'exact' ? 'in progress' : 'launching'}</span>{/if}{#if collection.queued}<span>{collection.queued} queued</span>{/if}{#if collection.failed}<span class="error">{collection.failed} need attention</span>{/if}</div>
|
||||
<div class="collection-meter" aria-hidden="true"><span class="verified" style:width={`${collection.done / (collection.total || 1) * 100}%`}></span><span class="unsuccessful" style:width={`${collection.failed / (collection.total || 1) * 100}%`}></span></div>
|
||||
<div class="collection-footer"><span class="meta">Queued {$timeFormat.dateTime(collection.created_at)}</span><span class="collection-link">{collection.active ? 'Follow progress' : 'View results'} →</span></div>
|
||||
<div class="row-head"><strong>{collection.name}</strong><Badge variant={launchVariant(collection)}>{launchLabel(collection)}</Badge></div>
|
||||
<div class="collection-counts"><span><strong>{collection.done} / {collection.total}</strong> {collection.mode === 'exact' ? 'completed' : 'ready'}</span>{#if collection.provisioning + collection.verifying}<span>{collection.provisioning + collection.verifying} {collection.mode === 'exact' ? 'in progress' : 'launching'}</span>{/if}{#if collection.queued}<span>{collection.queued} queued</span>{/if}{#if collection.failed}<span class="error">{collection.failed} need attention</span>{/if}{#if collection.cancelled}<span>{collection.cancelled} cancelled</span>{/if}{#if collection.cleanupActive}<span>{collection.cleanupActive} cleaning up</span>{/if}{#if collection.cleanupFailed}<span class="error">{collection.cleanupFailed} cleanup{collection.cleanupFailed === 1 ? ' needs' : 's need'} attention</span>{/if}</div>
|
||||
<div class="collection-meter" aria-hidden="true"><span class="verified" style:width={`${collection.done / (collection.total || 1) * 100}%`}></span><span class="unsuccessful" style:width={`${collection.failed / (collection.total || 1) * 100}%`}></span><span class="cancelled" style:width={`${collection.cancelled / (collection.total || 1) * 100}%`}></span></div>
|
||||
<div class="collection-footer"><span class="meta">Queued {$timeFormat.dateTime(collection.created_at)}</span><span class="collection-link">{inProgress(collection) ? 'Follow progress' : 'View results'} →</span></div>
|
||||
</a>
|
||||
{/each}</div>
|
||||
{:else if !launchError}
|
||||
<EmptyState title={tab === 'active' ? 'No collection launches in flight' : 'No collection launch history'} description={tab === 'active' ? 'Preview fleet changes from Your collections to start a launch.' : 'Completed launches and launches needing attention appear here.'} />
|
||||
<EmptyState title={tab === 'active' ? 'No collection launches in flight' : 'No collection launch history'} description={tab === 'active' ? 'Preview fleet changes from Your collections to start a launch.' : 'Completed and cancelled launches, including those needing attention, appear here.'} />
|
||||
{/if}
|
||||
<ListFooter total={visibleLaunches.length} bind:limit={launchLimit} onchange={() => saveContext()} step={10} label="launches" />
|
||||
</section>
|
||||
@@ -165,7 +178,7 @@
|
||||
.collection-counts { display: flex; flex-wrap: wrap; gap: 10px 20px; font-size: 13px; color: var(--text-secondary); }
|
||||
.collection-counts strong { color: var(--state-online); font-weight: 600; font-variant-numeric: tabular-nums; }
|
||||
.collection-meter { display: flex; height: 4px; background: var(--surface-raised); overflow: hidden; border-radius: var(--radius-pill); }
|
||||
.collection-meter .verified { background: var(--state-online); }.collection-meter .unsuccessful { background: var(--state-danger); }
|
||||
.collection-meter .verified { background: var(--state-online); }.collection-meter .unsuccessful { background: var(--state-danger); }.collection-meter .cancelled { background: var(--text-tertiary); }
|
||||
.collection-footer { display: flex; align-items: baseline; justify-content: space-between; flex-wrap: wrap; gap: 8px 16px; font-size: 12px; }
|
||||
.meta { color: var(--text-secondary); }.collection-link { color: var(--accent-primary-hover); }
|
||||
.blueprint p, .blueprint-empty p { color: var(--text-secondary); font-size: 13px; line-height: 1.6; }
|
||||
|
||||
@@ -3,23 +3,41 @@
|
||||
import BackLink from '$ui/components/BackLink.svelte';
|
||||
import { onMount } from 'svelte';
|
||||
import { pollVisible } from '$ui/poll.js';
|
||||
import { confirmAction } from '$ui/confirm.js';
|
||||
import ServerIdentity from '$ui/components/ServerIdentity.svelte';
|
||||
import HistoricalServerIdentity from '$ui/components/HistoricalServerIdentity.svelte';
|
||||
import { page } from '$app/stores';
|
||||
import SectionHeader from '$ui/components/primitives/SectionHeader.svelte';
|
||||
import Breadcrumbs from '$ui/components/primitives/Breadcrumbs.svelte';
|
||||
import Badge from '$ui/components/primitives/Badge.svelte';
|
||||
import Button from '$ui/components/primitives/Button.svelte';
|
||||
import type { CollectionDetail } from '$lib/deployment-plan.js';
|
||||
|
||||
type CollectionServer = CollectionDetail['servers'][number];
|
||||
let detail = $state<CollectionDetail | null>(null);
|
||||
let error = $state('');
|
||||
let cancelError = $state('');
|
||||
let cancelling = $state(false);
|
||||
let cancellationAccepted = $state(false);
|
||||
let requestNumber = 0;
|
||||
let collection = $derived(detail?.collection);
|
||||
const launching = (server: { state: string; server_deleted_at?: string | null }) => !server.server_deleted_at && ['running', 'awaiting-health'].includes(server.state);
|
||||
const needsAttention = (server: { state: string }) => ['failed', 'interrupted', 'warning', 'cancelled'].includes(server.state);
|
||||
const needsAttention = (server: { state: string }) => ['failed', 'interrupted', 'warning'].includes(server.state);
|
||||
const cleanupNeedsAttention = (cancellation: NonNullable<CollectionServer['cancellation']>) => needsAttention(cancellation) || cancellation.state === 'cancelled';
|
||||
const canOperate = $derived($page.data.user?.role === 'admin' || $page.data.user?.role === 'operator');
|
||||
const cancellationRequested = $derived(!!collection?.cancelled_at || cancellationAccepted);
|
||||
const inProgress = $derived(!!collection && (collection.active > 0 || collection.cleanupActive > 0));
|
||||
const exact = $derived(collection?.mode === 'exact');
|
||||
const currentServers = $derived(detail?.servers.filter(launching) ?? []);
|
||||
const heading = $derived.by(() => {
|
||||
if (!collection) return 'Collection launch';
|
||||
if (cancellationRequested) return inProgress ? 'Cancellation requested' : collection.failed || collection.cleanupFailed ? 'Cancellation needs attention' : 'Collection launch cancelled';
|
||||
if (collection.cleanupActive) return 'Cleaning up cancelled launches';
|
||||
if (collection.active) return exact ? 'Applying your collection' : currentServers.length ? 'Bringing your lineup online' : 'Your lineup is queued';
|
||||
if (collection.failed || collection.cleanupFailed) return exact ? 'Collection finished with issues' : 'Launch complete with issues';
|
||||
if (collection.cancelled) return 'Collection finished with cancellations';
|
||||
return exact ? 'Collection applied' : 'Your lineup is ready';
|
||||
});
|
||||
const otherLaunches = $derived(detail?.queue.active.filter(active => !detail?.servers.some(server => server.id === active.id)) ?? []);
|
||||
const firstQueued = $derived(detail?.servers.find(server => !server.server_deleted_at && server.state === 'queued'));
|
||||
const labels: Record<string, string> = { queued: 'Queued', running: 'Launching', 'awaiting-health': 'Checking readiness', done: 'Ready', warning: 'Needs review', failed: 'Failed', interrupted: 'Interrupted', waiting: 'Waiting', cancelled: 'Cancelled' };
|
||||
@@ -47,7 +65,33 @@
|
||||
if (current === requestNumber && id === $page.params.id) { detail = body; error = ''; }
|
||||
} catch (err) { if (current === requestNumber) error = (err as Error).message; }
|
||||
}
|
||||
$effect(() => { $page.params.id; detail = null; void load(); });
|
||||
async function cancelLaunch() {
|
||||
if (!collection || !detail?.canCancel || cancellationRequested || cancelling || !canOperate) return;
|
||||
const id = collection.id;
|
||||
const accepted = await confirmAction({
|
||||
title: 'Cancel collection launch?',
|
||||
message: 'Stops unfinished server creations, including eligible failed or interrupted launches, and deletes their containers and instance files. This can disconnect active players, even when player status is unknown. Shared game files are kept. Keys and ports are released only after removal is confirmed.\n\nCompleted launches and retained servers are kept. Pending removals are cancelled where safe; removals that have already started or are shared with another collection may still finish.',
|
||||
confirmLabel: 'Cancel launch and clean up',
|
||||
danger: true
|
||||
});
|
||||
if (!accepted || id !== $page.params.id || cancellationRequested || cancelling) return;
|
||||
cancelling = true;
|
||||
cancelError = '';
|
||||
try {
|
||||
const res = await fetch(`/api/deployment-collections/${encodeURIComponent(id)}/cancel`, { method: 'POST' });
|
||||
const body = await res.json();
|
||||
if (!res.ok) throw new Error(body.error || 'Could not cancel this collection launch.');
|
||||
if (id !== $page.params.id) return;
|
||||
cancellationAccepted = true;
|
||||
await load();
|
||||
} catch (err) {
|
||||
if (id === $page.params.id) {
|
||||
cancelError = (err as Error).message;
|
||||
await load();
|
||||
}
|
||||
} finally { if (id === $page.params.id) cancelling = false; }
|
||||
}
|
||||
$effect(() => { $page.params.id; detail = null; cancelError = ''; cancelling = false; cancellationAccepted = false; void load(); });
|
||||
onMount(() => { const stop = pollVisible(load, 5000, false); return () => { stop(); requestNumber++; }; });
|
||||
</script>
|
||||
|
||||
@@ -61,20 +105,35 @@
|
||||
{#if collection && detail}
|
||||
<section class="summary" aria-label="Collection progress">
|
||||
<div class="summary-heading">
|
||||
<div><span class="eyebrow">Collection progress</span><h2>{exact ? collection.active ? 'Applying your collection' : collection.failed ? 'Collection finished with issues' : 'Collection applied' : collection.active ? currentServers.length ? 'Bringing your lineup online' : 'Your lineup is queued' : collection.failed ? 'Launch complete with issues' : 'Your lineup is ready'}</h2></div>
|
||||
<span class="completion"><strong>{collection.done + collection.failed}</strong> / {collection.total} finished</span>
|
||||
<div><span class="eyebrow">Collection progress</span><h2>{heading}</h2></div>
|
||||
<span class="completion"><strong>{collection.done + collection.failed + collection.cancelled}</strong> / {collection.total} finished</span>
|
||||
</div>
|
||||
<div class="progress" role="progressbar" aria-label="Finished fleet operations" aria-valuemin={0} aria-valuemax={collection.total || 1} aria-valuenow={collection.done + collection.failed} aria-valuetext={`${collection.done} completed, ${collection.failed} need attention, ${collection.active} remaining`}>
|
||||
<div class="progress" role="progressbar" aria-label="Finished fleet operations" aria-valuemin={0} aria-valuemax={collection.total || 1} aria-valuenow={collection.done + collection.failed + collection.cancelled} aria-valuetext={`${collection.done} completed, ${collection.failed} need attention, ${collection.cancelled} cancelled, ${collection.active} remaining, ${collection.cleanupActive} cleaning up`}>
|
||||
<span class="progress-ready" style:width={`${collection.done / (collection.total || 1) * 100}%`}></span>
|
||||
<span class="progress-failed" style:width={`${collection.failed / (collection.total || 1) * 100}%`}></span>
|
||||
<span class="progress-cancelled" style:width={`${collection.cancelled / (collection.total || 1) * 100}%`}></span>
|
||||
</div>
|
||||
<dl class="counts">
|
||||
<dl class="counts" class:has-cancellations={cancellationRequested || collection.cancelled > 0}>
|
||||
<div class="ready-count"><dt>{exact ? 'Completed' : 'Ready'}</dt><dd>{collection.done}</dd></div>
|
||||
<div><dt>{exact ? 'In progress' : 'Launching'}</dt><dd>{collection.provisioning + collection.verifying}</dd></div>
|
||||
<div><dt>Queued</dt><dd>{collection.queued}</dd></div>
|
||||
<div class:attention-count={collection.failed > 0}><dt>Need attention</dt><dd>{collection.failed}</dd></div>
|
||||
{#if cancellationRequested || collection.cancelled > 0}<div><dt>Cancelled</dt><dd>{collection.cancelled}</dd></div>{/if}
|
||||
</dl>
|
||||
<p class="screen-reader" role="status">{collection.done} completed, {collection.provisioning + collection.verifying} in progress, {collection.queued} queued, {collection.failed} need attention.</p>
|
||||
<p class="screen-reader" role="status">{collection.done} completed, {collection.provisioning + collection.verifying} in progress, {collection.queued} queued, {collection.failed} need attention, {collection.cancelled} cancelled, {collection.cleanupActive} cleaning up, {collection.cleanupFailed} cleanups need attention.</p>
|
||||
{#if cancellationRequested}
|
||||
<p class="cancellation-note">Completed launches and retained servers are kept. Removals that already started or are shared with another collection may still finish. Follow each cancelled server's cleanup below until removal is confirmed.</p>
|
||||
{/if}
|
||||
{#if collection.cleanupActive || collection.cleanupFailed}
|
||||
<div class="cleanup-summary" role="status">
|
||||
{#if collection.cleanupActive}<p>{collection.cleanupActive} server cleanup{collection.cleanupActive === 1 ? '' : 's'} in progress. Keys and ports remain reserved until removal is confirmed.</p>{/if}
|
||||
{#if collection.cleanupFailed}<p class="error">{collection.cleanupFailed} cleanup{collection.cleanupFailed === 1 ? ' needs' : 's need'} attention. Open the cleanup operation below to review and recover.</p>{/if}
|
||||
</div>
|
||||
{/if}
|
||||
{#if canOperate && detail.canCancel && !cancellationRequested}
|
||||
<div class="cancel-action"><Button variant="danger" loading={cancelling} onclick={cancelLaunch}>Cancel collection launch</Button><p>Stops unfinished launches and removes their containers and instance files. Completed launches and retained servers are kept.</p></div>
|
||||
{/if}
|
||||
{#if cancelError}<p class="error" role="alert">{cancelError}</p>{/if}
|
||||
</section>
|
||||
|
||||
{#if currentServers.length}
|
||||
@@ -110,7 +169,7 @@
|
||||
<div class="queue-heading"><h2 id="queue-title">{exact ? 'Fleet changes' : 'Launch order'}</h2><span class="muted">{collection.total} operation{collection.total === 1 ? '' : 's'}</span></div>
|
||||
<ol class="servers">
|
||||
{#each detail.servers as server, index (server.id)}
|
||||
<li class:launching={launching(server)} class:attention={needsAttention(server)}>
|
||||
<li class:launching={launching(server)} class:attention={needsAttention(server) || (server.cancellation && cleanupNeedsAttention(server.cancellation))}>
|
||||
<a class="server" href={contextHref(`/provisioning/${server.id}`, $page.url)}>
|
||||
<span class="order" class:ready={server.state === 'done'} aria-hidden="true">{server.state === 'done' ? '✓' : String(index + 1).padStart(2, '0')}</span>
|
||||
<div class="server-content">
|
||||
@@ -119,6 +178,13 @@
|
||||
<div class="server-meta"><span>{server.action === 'destroy' ? 'Remove server' : server.port === null ? 'Create server · Port not assigned yet' : `Create server · Port ${server.port}`}</span><span>{needsAttention(server) ? 'Review & recover' : 'View operation'} <span aria-hidden="true">→</span></span></div>
|
||||
</div>
|
||||
</a>
|
||||
{#if server.cancellation}
|
||||
<a class="cleanup-link" class:error={cleanupNeedsAttention(server.cancellation)} href={contextHref(`/provisioning/${server.cancellation.cleanupId}`, $page.url)}>
|
||||
<strong>{server.cancellation.state === 'done' ? 'Cleanup completed' : cleanupNeedsAttention(server.cancellation) ? 'Cleanup needs attention' : 'Cleanup in progress'}</strong>
|
||||
<span>{server.cancellation.error ?? server.cancellation.stage}</span>
|
||||
<span class="cleanup-operation">{cleanupNeedsAttention(server.cancellation) ? 'Review cleanup' : 'View cleanup progress'} <span aria-hidden="true">→</span></span>
|
||||
</a>
|
||||
{/if}
|
||||
</li>
|
||||
{/each}
|
||||
</ol>
|
||||
@@ -131,10 +197,13 @@
|
||||
h2, p { margin: 0; }h2 { font-size: 21px; letter-spacing: var(--tracking-tight); }.eyebrow { display: block; font-size: 11px; text-transform: uppercase; letter-spacing: .1em; color: var(--text-secondary); margin-bottom: 9px; }
|
||||
a { color: var(--accent-primary-hover); }a:focus-visible, button:focus-visible { outline: 2px solid var(--border-focus); outline-offset: 3px; }.muted { color: var(--text-secondary); font-size: 12px; line-height: 1.6; }.error { color: var(--state-danger); }
|
||||
.summary { display: grid; gap: 24px; padding: 28px; border: 1px solid var(--border-strong); border-radius: var(--radius-lg); background: var(--surface); }.summary-heading, .queue-heading { display: flex; align-items: center; justify-content: space-between; gap: 16px; flex-wrap: wrap; }.completion { color: var(--text-secondary); font-size: 13px; white-space: nowrap; font-variant-numeric: tabular-nums; }.completion strong { color: var(--text-primary); font-weight: 600; }
|
||||
.progress { display: flex; height: 6px; background: var(--surface-raised); overflow: hidden; border-radius: var(--radius-pill); }.progress span { transition: width var(--dur-base) var(--ease-standard); }.progress-ready { background: var(--state-online); }.progress-failed { background: var(--state-danger); }
|
||||
.progress { display: flex; height: 6px; background: var(--surface-raised); overflow: hidden; border-radius: var(--radius-pill); }.progress span { transition: width var(--dur-base) var(--ease-standard); }.progress-ready { background: var(--state-online); }.progress-failed { background: var(--state-danger); }.progress-cancelled { background: var(--text-tertiary); }
|
||||
.counts.has-cancellations { grid-template-columns: repeat(5, minmax(0, 1fr)); }
|
||||
.cancellation-note, .cancel-action p, .cleanup-summary { font-size: 13px; line-height: 1.6; color: var(--text-secondary); }.cancel-action { display: grid; gap: 10px; justify-items: start; }.cleanup-summary { display: grid; gap: 8px; }
|
||||
.cleanup-link { display: grid; gap: 6px; margin: 0 24px 20px 68px; padding: 14px 16px; border: 1px solid var(--border-subtle); border-radius: var(--radius-sm); color: var(--text-secondary); text-decoration: none; font-size: 12px; line-height: 1.6; overflow-wrap: anywhere; }.cleanup-link:hover { background: var(--surface-raised); }.cleanup-link strong { color: var(--text-primary); }.cleanup-link.error, .cleanup-link.error strong { color: var(--state-danger); }.cleanup-operation { color: var(--accent-primary-hover); }
|
||||
.counts { display: grid; grid-template-columns: repeat(4, minmax(0, 1fr)); gap: 16px; margin: 0; }.counts > div + div { padding-left: 24px; border-left: 1px solid var(--border-subtle); }dt { color: var(--text-secondary); font-size: 12px; }dd { font-variant-numeric: tabular-nums; font-size: 28px; font-weight: 600; margin: 6px 0 0; letter-spacing: -.025em; }.ready-count dd { color: var(--state-online); }.attention-count dd { color: var(--state-danger); }
|
||||
.current-launches { display: grid; gap: 12px; }.current-launches > .eyebrow { margin: 0; color: var(--accent-primary-hover); }.current-launch { display: flex; align-items: center; gap: 16px; color: inherit; text-decoration: none; padding: 20px 24px; border: 1px solid color-mix(in srgb, var(--accent-primary) 35%, transparent); border-radius: var(--radius-md); background: color-mix(in srgb, var(--accent-primary) 7%, var(--surface)); }.current-launch > div { min-width: 0; flex: 1; }.current-launch strong { font-size: 16px; }.current-launch p { margin-top: 7px; color: var(--text-secondary); font-size: 13px; line-height: 1.6; overflow-wrap: anywhere; }.current-launch:hover { border-color: var(--accent-primary); }.operation-link { font-size: 12px; color: var(--accent-primary-hover); white-space: nowrap; }.activity { flex: 0 0 9px; height: 9px; border-radius: 50%; background: var(--accent-primary-hover); box-shadow: 0 0 0 5px color-mix(in srgb, var(--accent-primary) 12%, transparent); }
|
||||
.waiting-note { display: flex; gap: 16px; padding: 0 4px; }.waiting-icon { color: var(--text-secondary); font-size: 26px; line-height: 1; }.waiting-note strong { font-size: 14px; }.waiting-note p { margin-top: 6px; color: var(--text-secondary); font-size: 13px; line-height: 1.6; }.other-launches { display: grid; gap: 10px; margin-top: 12px; }.other-launches a { display: flex; align-items: baseline; gap: 10px; width: fit-content; font-size: 13px; text-decoration: none; }.other-launches a:hover { text-decoration: underline; }
|
||||
.queue { display: grid; gap: 16px; }.queue-heading h2 { font-size: 17px; }.servers { padding: 0; margin: 0; list-style: none; border: 1px solid var(--border-subtle); border-radius: var(--radius-md); overflow: hidden; background: var(--surface); }.servers li + li { border-top: 1px solid var(--border-subtle); }.servers li { position: relative; }.servers li.launching::before, .servers li.attention::before { content: ''; width: 2px; position: absolute; inset: 0 auto 0 0; background: var(--accent-primary); pointer-events: none; }.servers li.attention::before { background: var(--state-danger); }.server { display: flex; gap: 20px; padding: 22px 24px; color: inherit; text-decoration: none; }.server:hover { background: var(--surface-raised); }.server:focus-visible { outline-offset: -3px; }.order { padding-top: 2px; color: var(--text-tertiary); font-size: 12px; font-family: var(--font-mono); flex: 0 0 24px; }.order.ready { color: var(--state-online); font-family: var(--font-body); font-size: 17px; padding-top: 0; }.server-content { flex: 1; min-width: 0; }.server-heading { display: flex; justify-content: space-between; align-items: flex-start; gap: 12px; flex-wrap: wrap; }.server-heading strong { font-size: 14px; }.server p { margin-top: 9px; color: var(--text-secondary); font-size: 12px; line-height: 1.6; overflow-wrap: anywhere; }.server p.error { color: var(--state-danger); }.server-meta { display: flex; justify-content: space-between; gap: 12px; flex-wrap: wrap; margin-top: 12px; color: var(--text-tertiary); font-size: 11px; }.server-meta span:last-child { color: var(--accent-primary-hover); }.screen-reader { position: absolute; width: 1px; height: 1px; padding: 0; overflow: hidden; clip-path: inset(50%); white-space: nowrap; border: 0; }
|
||||
@media (max-width: 600px) { .page { gap: 24px; }.summary { padding: 20px; gap: 20px; }h2 { font-size: 19px; }.counts { grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 20px; }.counts > div + div { padding-left: 0; border-left: 0; }.counts > div:nth-child(even) { padding-left: 20px; border-left: 1px solid var(--border-subtle); }dd { font-size: 25px; }.current-launch { padding: 20px; flex-wrap: wrap; gap: 12px; }.current-launch > div { flex-basis: calc(100% - 24px); }.operation-link { margin-left: 21px; }.server { gap: 12px; padding: 20px 16px; }.order { flex-basis: 20px; }.server-heading { row-gap: 10px; }.server-heading strong { flex-basis: 100%; }.completion { font-size: 12px; } }
|
||||
@media (max-width: 600px) { .page { gap: 24px; }.summary { padding: 20px; gap: 20px; }h2 { font-size: 19px; }.counts, .counts.has-cancellations { grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 20px; }.counts > div + div { padding-left: 0; border-left: 0; }.counts > div:nth-child(even) { padding-left: 20px; border-left: 1px solid var(--border-subtle); }dd { font-size: 25px; }.current-launch { padding: 20px; flex-wrap: wrap; gap: 12px; }.current-launch > div { flex-basis: calc(100% - 24px); }.operation-link { margin-left: 21px; }.server { gap: 12px; padding: 20px 16px; }.order { flex-basis: 20px; }.server-heading { row-gap: 10px; }.server-heading strong { flex-basis: 100%; }.completion { font-size: 12px; }.cleanup-link { margin: 0 16px 16px; } }
|
||||
</style>
|
||||
|
||||
@@ -24,6 +24,7 @@ describe('management route permissions', () => {
|
||||
['/api/servers', 'POST'], ['/api/servers/[name]', 'DELETE'], ['/api/provisioning', 'POST'],
|
||||
['/api/provisioning/[id]/cancel', 'POST'], ['/api/provisioning/[id]/resume', 'POST'],
|
||||
['/api/deployment-collections', 'POST'], ['/api/deployment-collections/preview', 'POST'],
|
||||
['/api/deployment-collections/[id]/cancel', 'POST'],
|
||||
['/api/blueprints', 'POST'], ['/api/blueprints/[id]', 'PUT'], ['/api/blueprints/[id]', 'DELETE'],
|
||||
['/api/betas', 'POST'], ['/api/betas/[id]/promote', 'POST'], ['/api/betas/[id]/rollback', 'POST'],
|
||||
['/api/campaigns/[id]/rollback', 'POST'], ['/api/deployments/[id]/rollback', 'POST'],
|
||||
|
||||
@@ -0,0 +1,283 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { newFixture, type Fixture } from '../helpers/fixture.js';
|
||||
import { prepareNativeHost } from '../helpers/native-host.js';
|
||||
import type { CollectionMode } from '../../src/lib/deployment-plan.js';
|
||||
|
||||
vi.mock('../../src/lib/server/integrations/rcon.js', () => ({
|
||||
rconGetPlayerCount: vi.fn(async () => 0),
|
||||
rconPlutoniumVersion: vi.fn(async () => ({ revision: 5354, error: null })),
|
||||
rconStatusResult: vi.fn(async () => ({ ok: true, endpoint: { host: 'agent', port: 28960 }, rawReply: '',
|
||||
status: { players: 0, map: 'zombie_theater', gametype: 'zom', roster: [] } }))
|
||||
}));
|
||||
vi.mock('../../src/lib/server/integrations/plutonium.js', () => ({
|
||||
checkRelease: vi.fn(async () => {}), freshRelease: () => 5354, probeVersion: vi.fn(async () => 5354)
|
||||
}));
|
||||
vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}), kickProvisioning: vi.fn(async () => {}) }));
|
||||
|
||||
let fixture: Fixture;
|
||||
let hostId: string;
|
||||
let launchNumber: number;
|
||||
|
||||
beforeEach(async () => {
|
||||
vi.resetModules(); vi.clearAllMocks();
|
||||
launchNumber = 0;
|
||||
fixture = await newFixture([]);
|
||||
vi.stubEnv('RCON_PASSWORD', 'private-rcon-password');
|
||||
vi.stubEnv('DEPLOYMENT_CREATE_CONCURRENCY', '1');
|
||||
hostId = (await prepareNativeHost()).host.id;
|
||||
const backend = await import('../../src/lib/server/backends/native.js');
|
||||
vi.spyOn(backend, 'nativeRequest').mockImplementation(async (_instance, kind) => {
|
||||
if (kind !== 'instance.inspect') throw new Error(`Unexpected operation: ${kind}`);
|
||||
return { healthy: true, state: 'running', exists: true } as never;
|
||||
});
|
||||
});
|
||||
afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); });
|
||||
|
||||
async function launch(count = 2, mapSlug = 'theater', mode: CollectionMode = 'fill', savedKeys = false) {
|
||||
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
|
||||
const { previewCollection, queueCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
|
||||
const blueprint = saveBlueprint({ name: `Cancellation fixture ${++launchNumber}`, entries: [{ game: 't5', mapSlug, count }] });
|
||||
const preview = previewCollection(blueprint.id, mode);
|
||||
const keys = Object.fromEntries(preview.servers.map(server => [server.name!, `${blueprint.id}-${server.name}`]));
|
||||
const queued = queueCollection({ previewId: preview.id, ...(savedKeys ? {} : { keys }) });
|
||||
if (!queued.collectionId) throw new Error('Expected collection work');
|
||||
return getCollection(queued.collectionId)!;
|
||||
}
|
||||
|
||||
async function advance() {
|
||||
const { processNativeProvisioning } = await import('../../src/lib/server/provisioning/native.js');
|
||||
await processNativeProvisioning(() => null);
|
||||
}
|
||||
|
||||
async function operationFor(deploymentId: string) {
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
const row = db().prepare('SELECT operation_id FROM native_provisioning WHERE deployment_id=?').get(deploymentId) as { operation_id: string | null } | undefined;
|
||||
const { getHostOperation } = await import('../../src/lib/server/hosts/index.js');
|
||||
return row?.operation_id ? getHostOperation(row.operation_id)! : null;
|
||||
}
|
||||
|
||||
async function acknowledge(deploymentId: string) {
|
||||
const operation = await operationFor(deploymentId);
|
||||
if (!operation) throw new Error('Missing durable native operation');
|
||||
const { reportHostOperation } = await import('../../src/lib/server/hosts/index.js');
|
||||
reportHostOperation(hostId, operation.id, { state: 'completed', result: { state: 'running', healthy: true } });
|
||||
await advance();
|
||||
}
|
||||
|
||||
describe('collection cancellation', () => {
|
||||
it('cancels every queued member, releases saved keys, and remains idempotent after reopening the database', async () => {
|
||||
const { saveServerKey, serverKeyPool } = await import('../../src/lib/server/fleet/server-keys.js');
|
||||
for (const key of ['first', 'second', 'third']) saveServerKey('t5', key);
|
||||
const launchDetail = await launch(3, 'theater', 'fill', true);
|
||||
const { cancelCollection, getCollection, listCollections } = await import('../../src/lib/server/fleet/collections.js');
|
||||
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||
expect(launchDetail.canCancel).toBe(true);
|
||||
expect(serverKeyPool('t5')).toEqual({ total: 3, available: 0 });
|
||||
expect(cancelCollection(launchDetail.collection.id)).toBe(true);
|
||||
const cancelled = getCollection(launchDetail.collection.id)!;
|
||||
expect(cancelled.canCancel).toBe(false);
|
||||
expect(cancelled.collection).toMatchObject({ cancelled_at: expect.any(String), total: 3, cancelled: 3,
|
||||
done: 0, active: 0, failed: 0, cleanupActive: 0, cleanupFailed: 0 });
|
||||
expect(cancelled.servers.every(server => server.state === 'cancelled' && server.cancellation?.state === 'done')).toBe(true);
|
||||
expect(serverKeyPool('t5')).toEqual({ total: 3, available: 3 });
|
||||
expect(db().prepare('SELECT * FROM server_key_leases').all()).toEqual([]);
|
||||
closeDb();
|
||||
expect(cancelCollection(launchDetail.collection.id)).toBe(true);
|
||||
expect(getCollection(launchDetail.collection.id)).toEqual(cancelled);
|
||||
expect(listCollections()[0]).toEqual(cancelled.collection);
|
||||
expect(db().prepare('SELECT * FROM native_creation_cancellations').all()).toHaveLength(3);
|
||||
await advance();
|
||||
expect(db().prepare("SELECT * FROM ec_host_operations WHERE kind LIKE 'instance.%'").all()).toEqual([]);
|
||||
});
|
||||
|
||||
it('rolls back all child cancellation and key releases if one cancellation fails', async () => {
|
||||
const detail = await launch(2);
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
const { cancelCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
|
||||
db().exec(`CREATE TRIGGER reject_second_cancellation BEFORE UPDATE OF state ON server_deployments
|
||||
WHEN NEW.state='cancelled' AND OLD.id='${detail.servers[1].id}'
|
||||
BEGIN SELECT RAISE(ABORT, 'Cancellation fixture failure'); END`);
|
||||
expect(() => cancelCollection(detail.collection.id)).toThrow('Cancellation fixture failure');
|
||||
expect(getCollection(detail.collection.id)?.collection.cancelled_at).toBeNull();
|
||||
expect(getCollection(detail.collection.id)?.servers.map(server => server.state)).toEqual(['queued', 'queued']);
|
||||
expect(db().prepare('SELECT * FROM native_creation_cancellations').all()).toEqual([]);
|
||||
expect(db().prepare('SELECT * FROM server_key_leases').all()).toHaveLength(2);
|
||||
expect(db().prepare("SELECT * FROM server_deployments WHERE action='destroy'").all()).toEqual([]);
|
||||
db().exec('DROP TRIGGER reject_second_cancellation');
|
||||
expect(cancelCollection(detail.collection.id)).toBe(true);
|
||||
});
|
||||
|
||||
it('cleans up unfinished members that are waiting for health, interrupted, or failed', async () => {
|
||||
const detail = await launch(4);
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
const { cancelCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
|
||||
const { serverKeyPool } = await import('../../src/lib/server/fleet/server-keys.js');
|
||||
['waiting', 'awaiting-health', 'interrupted', 'failed'].forEach((state, index) => {
|
||||
db().prepare('UPDATE server_deployments SET state=?,error=? WHERE id=?').run(state, 'Previous launch problem', detail.servers[index].id);
|
||||
});
|
||||
expect(getCollection(detail.collection.id)?.collection.failed).toBe(2);
|
||||
expect(cancelCollection(detail.collection.id)).toBe(true);
|
||||
const result = getCollection(detail.collection.id)!;
|
||||
expect(result.collection).toMatchObject({ total: 4, cancelled: 4, active: 0, failed: 0 });
|
||||
expect(result.servers.every(server => server.state === 'cancelled' && server.error === null && server.cancellation?.state === 'done')).toBe(true);
|
||||
expect(serverKeyPool('t5').available).toBe(4);
|
||||
});
|
||||
|
||||
it('preserves successful members, retained servers, and another launch', async () => {
|
||||
const retained = await launch(1);
|
||||
await advance(); await acknowledge(retained.servers[0].id);
|
||||
const detail = await launch(4);
|
||||
const unrelated = await launch(1, 'moon');
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
const { cancelCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
|
||||
// Completed jobs, including successful jobs with warnings, are never teardown candidates.
|
||||
db().prepare("UPDATE server_deployments SET state='done' WHERE id=?").run(detail.servers[0].id);
|
||||
db().prepare("UPDATE server_deployments SET state='warning' WHERE id=?").run(detail.servers[1].id);
|
||||
expect(detail.retainedServers).toEqual([retained.servers[0].server]);
|
||||
expect(cancelCollection(detail.collection.id)).toBe(true);
|
||||
const result = getCollection(detail.collection.id)!;
|
||||
expect(result.collection).toMatchObject({ done: 2, cancelled: 1, failed: 0 });
|
||||
expect(result.servers.map(server => server.state)).toEqual(['done', 'warning', 'cancelled']);
|
||||
expect(result.servers.slice(0, 2).every(server => server.cancellation === null)).toBe(true);
|
||||
expect(getCollection(retained.collection.id)?.collection).toMatchObject({ cancelled_at: null, done: 1 });
|
||||
expect(db().prepare('SELECT name FROM servers WHERE name=?').get(retained.servers[0].server)).toEqual({ name: retained.servers[0].server });
|
||||
expect(getCollection(unrelated.collection.id)?.collection).toMatchObject({ cancelled_at: null, active: 1 });
|
||||
});
|
||||
|
||||
it.each(['completed', 'failed'] as const)('tracks running member cleanup through a %s agent result', async outcome => {
|
||||
const detail = await launch(2);
|
||||
await advance();
|
||||
const original = (await operationFor(detail.servers[0].id))!;
|
||||
expect(original.kind).toBe('instance.create');
|
||||
const { cancelCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
const { reportHostOperation } = await import('../../src/lib/server/hosts/index.js');
|
||||
const { serverKeyPool } = await import('../../src/lib/server/fleet/server-keys.js');
|
||||
expect(cancelCollection(detail.collection.id)).toBe(true);
|
||||
const cancelled = getCollection(detail.collection.id)!;
|
||||
expect(cancelled.collection).toMatchObject({ active: 0, failed: 0, cancelled: 2, cleanupActive: 1, cleanupFailed: 0 });
|
||||
const cleanup = cancelled.servers[0].cancellation!;
|
||||
expect(cleanup).toMatchObject({ cleanupId: expect.any(String), state: 'running', error: null });
|
||||
const command = (await operationFor(cleanup.cleanupId))!;
|
||||
expect(command.kind).toBe('instance.cancel-create');
|
||||
expect(command.payload).toEqual({ createOperationId: original.id });
|
||||
expect(serverKeyPool('t5').available).toBe(1);
|
||||
expect(db().prepare('SELECT * FROM ec_host_ports').all()).toHaveLength(2);
|
||||
reportHostOperation(hostId, command.id, outcome === 'completed'
|
||||
? { state: 'completed', result: { exists: false } }
|
||||
: { state: 'failed', error: 'Agent refused cleanup before mutation' });
|
||||
await advance();
|
||||
const settled = getCollection(detail.collection.id)!;
|
||||
expect(settled.collection).toMatchObject({ active: 0, failed: 0, cancelled: 2, cleanupActive: 0, cleanupFailed: outcome === 'failed' ? 1 : 0 });
|
||||
expect(settled.servers[0].cancellation).toMatchObject({ cleanupId: cleanup.cleanupId,
|
||||
state: outcome === 'completed' ? 'done' : 'failed', error: outcome === 'completed' ? null : 'Agent refused cleanup before mutation' });
|
||||
expect(serverKeyPool('t5').available).toBe(outcome === 'completed' ? 2 : 1);
|
||||
expect(db().prepare('SELECT * FROM ec_host_ports').all()).toHaveLength(outcome === 'completed' ? 0 : 2);
|
||||
expect(cancelCollection(detail.collection.id)).toBe(true);
|
||||
expect(db().prepare('SELECT * FROM native_creation_cancellations').all()).toHaveLength(2);
|
||||
});
|
||||
|
||||
it('skips replaced instances and preserves cleanup already requested for an individual member', async () => {
|
||||
const detail = await launch(3);
|
||||
await advance();
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
const { cancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
|
||||
const { cancelCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
|
||||
db().prepare('UPDATE native_instances SET generation=generation+1 WHERE server=?').run(detail.servers[0].server);
|
||||
const independentCleanup = cancelCreation(detail.servers[1].id)!;
|
||||
expect(independentCleanup).toBeTruthy();
|
||||
expect(cancelCollection(detail.collection.id)).toBe(true);
|
||||
const result = getCollection(detail.collection.id)!;
|
||||
expect(result.servers[0]).toMatchObject({ state: 'running', cancellation: null });
|
||||
expect(result.servers[1].cancellation?.cleanupId).toBe(independentCleanup);
|
||||
expect(result.servers[2]).toMatchObject({ state: 'cancelled', cancellation: { state: 'done' } });
|
||||
expect(db().prepare('SELECT * FROM native_creation_cancellations').all()).toHaveLength(2);
|
||||
expect(db().prepare('SELECT generation,retired_at FROM native_instances WHERE server=?').get(detail.servers[0].server))
|
||||
.toEqual({ generation: 2, retired_at: null });
|
||||
});
|
||||
|
||||
it('requests cleanup for a member whose individual job was cancelled without removing its instance', async () => {
|
||||
const detail = await launch(1);
|
||||
await advance();
|
||||
const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
|
||||
const { cancelCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
|
||||
const { serverKeyPool } = await import('../../src/lib/server/fleet/server-keys.js');
|
||||
expect(cancelProvisioningJob(detail.servers[0].id)).toBe(true);
|
||||
expect(serverKeyPool('t5').available).toBe(0);
|
||||
expect(getCollection(detail.collection.id)?.canCancel).toBe(true);
|
||||
expect(cancelCollection(detail.collection.id)).toBe(true);
|
||||
const result = getCollection(detail.collection.id)!;
|
||||
expect(result.collection).toMatchObject({ cancelled: 1, cleanupActive: 1 });
|
||||
const cleanup = result.servers[0].cancellation!;
|
||||
expect(await operationFor(cleanup.cleanupId)).toMatchObject({ kind: 'instance.cancel-create' });
|
||||
await acknowledge(cleanup.cleanupId);
|
||||
expect(serverKeyPool('t5').available).toBe(1);
|
||||
expect(getCollection(detail.collection.id)?.collection.cleanupActive).toBe(0);
|
||||
});
|
||||
|
||||
it.each(['waiting', 'shared', 'started', 'manual'] as const)('handles an exact collection with a %s removal', async removalState => {
|
||||
const retained = await launch(1);
|
||||
await advance(); await acknowledge(retained.servers[0].id);
|
||||
if (removalState === 'manual') {
|
||||
const { destroyServer } = await import('../../src/lib/server/provisioning/instances.js');
|
||||
expect(destroyServer(retained.servers[0].server).ok).toBe(true);
|
||||
}
|
||||
const detail = await launch(1, 'moon', 'exact');
|
||||
const removal = detail.servers.find(server => server.action === 'destroy')!;
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
const { cancelCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
|
||||
if (removalState === 'shared') {
|
||||
db().prepare('INSERT INTO server_launch_collections(id,name,created_at,mode) VALUES(?,?,?,?)')
|
||||
.run('other-collection', 'Another exact launch', new Date().toISOString(), 'exact');
|
||||
db().prepare('INSERT INTO server_collection_removals(collection_id,deployment_id,position) VALUES(?,?,0)')
|
||||
.run('other-collection', removal.id);
|
||||
} else if (removalState === 'started') {
|
||||
await advance();
|
||||
expect(await operationFor(removal.id)).toMatchObject({ kind: 'instance.remove' });
|
||||
}
|
||||
expect(cancelCollection(detail.collection.id)).toBe(true);
|
||||
const result = getCollection(detail.collection.id)!;
|
||||
expect(result.servers.find(server => server.id === removal.id)?.state).toBe(removalState === 'waiting' ? 'cancelled' : removalState === 'started' ? 'running' : 'waiting');
|
||||
expect(result.servers.find(server => server.action === 'create')?.state).toBe('cancelled');
|
||||
expect(db().prepare('SELECT name FROM servers WHERE name=?').get(removal.server)).toEqual({ name: removal.server });
|
||||
if (removalState === 'shared') {
|
||||
expect(getCollection('other-collection')?.canCancel).toBe(true);
|
||||
expect(cancelCollection('other-collection')).toBe(true);
|
||||
expect(getCollection('other-collection')?.servers[0].state).toBe('cancelled');
|
||||
}
|
||||
});
|
||||
|
||||
it('returns API errors for missing or completed collections and accepts repeated cancellations', async () => {
|
||||
const completed = await launch(1);
|
||||
await advance(); await acknowledge(completed.servers[0].id);
|
||||
const { cancelCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
|
||||
const { POST } = await import('../../src/routes/api/deployment-collections/[id]/cancel/+server.js');
|
||||
const { kickProvisioning } = await import('../../src/lib/server/pipeline/worker.js');
|
||||
expect(cancelCollection('missing')).toBe(false);
|
||||
expect(cancelCollection(completed.collection.id)).toBe(false);
|
||||
expect(getCollection(completed.collection.id)?.canCancel).toBe(false);
|
||||
expect((await POST({ params: { id: 'missing' } })).status).toBe(404);
|
||||
expect((await POST({ params: { id: completed.collection.id } })).status).toBe(409);
|
||||
expect(kickProvisioning).not.toHaveBeenCalled();
|
||||
const pending = await launch(2);
|
||||
for (let repeat = 0; repeat < 2; repeat++) {
|
||||
const response = await POST({ params: { id: pending.collection.id } });
|
||||
expect(response.status).toBe(200);
|
||||
expect(await response.json()).toEqual({ ok: true });
|
||||
}
|
||||
expect(kickProvisioning).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it('migrates existing collections with no cancellation marker and preserves their jobs', async () => {
|
||||
const detail = await launch(1);
|
||||
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||
db().exec('ALTER TABLE server_launch_collections DROP COLUMN cancelled_at');
|
||||
closeDb();
|
||||
const { getCollection, cancelCollection } = await import('../../src/lib/server/fleet/collections.js');
|
||||
const migrated = getCollection(detail.collection.id)!;
|
||||
expect(migrated.collection.cancelled_at).toBeNull();
|
||||
expect(migrated.canCancel).toBe(true);
|
||||
expect(migrated.servers[0]).toMatchObject({ id: detail.servers[0].id, state: 'queued', cancellation: null });
|
||||
expect(cancelCollection(detail.collection.id)).toBe(true);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user