Cancel active provisioning jobs when rolling back betas

docs/provisioning.md

@@ -154,6 +154,20 @@ heartbeat or completion-time promises.
 - `/api/provisioning/:id`, `/:id/resume`, `/:id/cancel`, `/preview`, and
   `/house-party` preserve the existing operation, recovery, removal cancellation,
   launch preview, and fleet destruction APIs.
+- `POST /api/provisioning/:id/cancel` cancels any active create, start, restart,
+  rebuild, stop, or destroy job, including health verification. It releases that
+  job's recovery capacity and prevents further steps; completed effects are not
+  undone. Undelivered host commands are withdrawn. Commands already delivered
+  retain their receipts, and replacements wait for those mutations to settle.
+  Cancelling is idempotent. Lists and details expose `canCancel`.
+- `POST /api/provisioning/:id/cancel-creation` retains the explicit native creation
+  cleanup action: cancel the launch and remove its partial instance. Cancelling
+  a job alone retains its existing instance/files for inspection or later cleanup.
+- Beta rollback cancels conflicting provisioning jobs on its original servers
+  before queuing independent restoration restarts. It does not wait for the
+  superseded restart's health verification. Native activation receipts must still
+  provide durable before images; rollback reconciles completed receipts itself.
+  Manual rollback keeps the replacement jobs held until the campaign is resumed.
 - `GET /api/provisioning/:id?events=recent` returns the same detail with five recent
   events plus activity totals/latest ID and a failure excerpt. Omitting the option
   preserves the full legacy event response. Progress, capacity, named related work,

docs/ui-ux-implementation.md

@@ -14,7 +14,7 @@ The changes prioritize clear operational outcomes, reviewed action scope, recove
 | 4. Beta reading order | Outcome, blockers, deadline, actions, and server results precede evidence and activity. Shared activity history is collapsed, searchable, ordered, and progressively displayed. |
 | 5. Beta recovery | Deadlines and pause/retry consequences are explicit. A fresh attempt confirms evidence reset and restarts; expired tests explain rollback and plan reuse. Sign-off availability follows test/run readiness. |
 | 6. Launch input retention | Refreshing previews retains entered keys only for unchanged exact server IDs. Changed assignments are explained. Keys remain in memory; managing keys in another tab preserves the launch context. |
-| 7. Removal cancellation | Waiting removals can be withdrawn from operation lists/details. A database transition competes atomically with the worker claim before external deletion. Once removal or confirmed-deletion cleanup begins, cancellation is unavailable. |
+| 7. Provisioning cancellation | Active jobs can be cancelled from operation lists/details, including restarts and health verification. Remaining steps stop; effects already performed are retained. Rollback supersedes conflicting jobs with fresh restoration requests. |
 | 8. Partial outcomes | Reconciliation reports updated and skipped servers separately, with reasons and follow-up links. All-skipped work is an error outcome. Rollback feedback stays beside its initiating operation. |
 | 9. Scope clarity | Scope editing previews added/removed eligible servers. Script/plugin unscoping explains retained live copies and tracking detachment; configs explain composition impact. Completion links to reconciliation. Eligibility, promoted versions, and installed inventory have separate labels. |
 | 10. Accessible controls | Busy buttons retain their accessible names. Field labels focus the associated native control. Consequential errors use alert semantics; completion uses status semantics. Campaign server states have accessible descriptions. |
@@ -43,7 +43,7 @@ Activity and notifications have been removed. Copy results appear next to their
 
 - Promotion uses `GET /api/betas/:id/promote` to review impact and `POST` with the returned token as `previewToken` to apply it. A changed plan returns HTTP 409 and needs renewed review.
 - Reconciliation uses `POST /api/drift` with `dryRun: true` and optional `servers`, then executes with the returned `expectedFingerprint`. Omitted servers explicitly mean the entire fleet. A stale review returns HTTP 409; changes detected during application are reported as skipped files.
-- `POST /api/deployments/:id/cancel` only withdraws a waiting removal. It does not restore a server after deletion has started.
+- `POST /api/deployments/:id/cancel` cancels remaining provisioning work, including restarts and health checks. It does not undo changes already made.
 - System status exposes no credential values. Its optional connection check verifies Komodo network reachability, not credential validity. Remote game-file volumes remain explicitly unverified.
 - Drafts contain nonsecret beta setup. Launch keys are not stored in browser drafts.
 - Log search/export covers the loaded tail, not an entire remote file or a full diagnostic archive.

src/lib/provisioning-detail.ts

@@ -95,6 +95,7 @@ export interface ProvisioningCapacity {
     }[];
 }
 export interface ProvisioningDetail {
+    canCancel?: boolean;
     canCancelCreation?: boolean;
     cancellation?: { cleanupId: string; state: string; stage: string; error: string | null } | null;
     backend?: 'native' | 'komodo';

src/lib/provisioning-list.ts

@@ -3,7 +3,7 @@ import type { ProvisioningDeployment } from './provisioning-detail.js';
 
 export type ProvisioningListRow = Pick<ProvisioningDeployment,
     'id' | 'server' | 'server_deleted_at' | 'action' | 'state' | 'stage' | 'started_at' |
-    'finished_at' | 'triggered_by' | 'error' | 'warnings' | 'server_exists'> & { canCancelRemoval: number };
+    'finished_at' | 'triggered_by' | 'error' | 'warnings' | 'server_exists'> & { canCancelRemoval: number; canCancel?: number };
 
 export interface ProvisioningListOptions {
     view?: 'active' | 'history' | 'all';

src/lib/server/fleet/beta.ts

@@ -13,6 +13,7 @@ async function exclusive<T>(id: number, action: () => Promise<T>): Promise<T> {
 import { mkdir, rm, writeFile } from 'node:fs/promises';
 import { readFile, exists } from '../backends/files.js';
 import { getHostOperation } from '../hosts/index.js';
+import { ensureHostSchema } from '../hosts/schema.js';
 
 import { join, posix } from 'node:path';
 import { betaArchiveOf, configFileOf, destOfKind, type ArtifactKind } from '../config.js';
@@ -35,6 +36,9 @@ import { composeServerCfg } from './config-compose.js';
 import { getServerScript, removeScript } from './inventory.js';
 import { cancelStagedFiles, isFileApplied, queueFileChange } from './staged-files.js';
 import { emit } from '../pipeline/bus.js';
+import { cancelProvisioningJob } from '../provisioning/cancellation.js';
+import { reconcileNativeProvisioningReceipt } from '../provisioning/native.js';
+import { requestRestart } from '../provisioning/requests.js';
 import type { BetaStage } from '../pipeline/bus.js';
 
 // Whether an artifact writes a dedicated on-disk file (script/plugin) or
@@ -573,7 +577,6 @@ export async function rollbackBeta(betaId: number, options: { restart?: 'manual'
     return exclusive(betaId, () => rollbackBetaInternal(betaId, options));
 }
 async function rollbackBetaInternal(betaId: number, options: { restart?: 'manual' | 'when-empty' }): Promise<RollbackResult> {
-    assertNativeActivationSettled(betaId);
     const beta = getBeta(betaId);
     if (!beta) throw new BetaError(`no such beta (id=${betaId})`, 'no-such-beta');
     if (beta.state !== 'running' && beta.state !== 'ready-promote') {
@@ -588,6 +591,20 @@ async function rollbackBetaInternal(betaId: number, options: { restart?: 'manual
         versionId: br.script_version_id
     }));
 
+    ensureHostSchema(db());
+    db().transaction(() => {
+        const jobs = db().prepare(`SELECT s.operation_id FROM provisioning_sources s WHERE s.kind='beta' AND s.source_id=?`)
+            .all(String(betaId)) as { operation_id: string }[];
+        for (const job of jobs) {
+            const undelivered = db().prepare(`SELECT 1 FROM native_provisioning n JOIN ec_host_operations o ON o.id=n.operation_id
+                WHERE n.deployment_id=? AND o.state='pending' AND o.last_delivered_at IS NULL`).get(job.operation_id);
+            if (undelivered) cancelProvisioningJob(job.operation_id, `Superseded by rollback of beta #${betaId}`);
+            reconcileNativeProvisioningReceipt(job.operation_id);
+        }
+        // Before images remain mandatory for a mutation already delivered to a host.
+        // Completed commands can be reconciled here without waiting for game health.
+        assertNativeActivationSettled(betaId);
+    })();
     const servers = listBetaServers(betaId);
     const restored = new Set<string>();
     const dropped = new Set<string>();
@@ -640,6 +657,12 @@ async function rollbackBetaInternal(betaId: number, options: { restart?: 'manual
 
     const deploymentId = restorations.length ? `dep-beta-rollback-${betaId}-${Date.now()}` : null;
     db().transaction(() => {
+        for (const bs of servers) {
+            if (!isActiveBetaServer(betaId, bs.server)) continue;
+            const pending = db().prepare(`SELECT id FROM server_deployments WHERE server=? AND server_deleted_at IS NULL
+                AND state IN ('queued','waiting','running','awaiting-health','interrupted')`).all(bs.server) as { id: string }[];
+            for (const job of pending) cancelProvisioningJob(job.id, `Superseded by rollback of beta #${betaId}`);
+        }
         cancelStagedFiles('beta', String(betaId));
         if (deploymentId) {
             db().prepare("INSERT INTO deployments (id, action, started_at, triggered_by, beta_id) VALUES (?, 'rollback', ?, ?, ?)")
@@ -661,6 +684,11 @@ async function rollbackBetaInternal(betaId: number, options: { restart?: 'manual
                     deploymentFileId: Number(file.lastInsertRowid) });
             }
         }
+        if (deploymentId) {
+            for (const server of new Set(restorations.map(change => change.server))) {
+                requestRestart(server, { kind: 'campaign', id: deploymentId });
+            }
+        }
         // The queued restoration owns a durable copy before snapshots are released.
         db().prepare('DELETE FROM beta_file_snapshots WHERE beta_id = ?').run(betaId);
         db().prepare("UPDATE betas SET state = 'rolled-back', closed_at = ? WHERE id = ?").run(nowIso(), betaId);

src/lib/server/provisioning/cancel-creation.ts

@@ -13,7 +13,7 @@ export function canCancelCreation(id: string): boolean {
             AND EXISTS (SELECT 1 FROM native_creation_intents intent WHERE intent.deployment_id=d.id)
             AND NOT EXISTS (SELECT 1 FROM native_instances replacement WHERE replacement.server=d.server AND replacement.retired_at IS NULL)
             AND NOT EXISTS (SELECT 1 FROM servers replacement WHERE replacement.name=d.server AND replacement.destroyed_at IS NULL)))
-        AND d.state IN ('queued','waiting','running','awaiting-health','interrupted','failed')
+        AND d.state IN ('queued','waiting','running','awaiting-health','interrupted','failed','cancelled')
         AND NOT EXISTS (SELECT 1 FROM server_deployments other WHERE other.server=d.server
             AND other.server_deleted_at IS NULL AND other.rowid>d.rowid AND other.state!='cancelled'
             AND NOT (other.action='destroy' AND other.state='waiting'

src/lib/server/provisioning/cancellation.ts

@@ -0,0 +1,59 @@
+import { db } from '../db/index.js';
+import { ensureHostSchema } from '../hosts/schema.js';
+import { recordProgress } from './progress.js';
+
+export function provisioningCancelled(id: string): boolean {
+    return !!db().prepare("SELECT 1 FROM server_deployments WHERE id=? AND state='cancelled'").get(id);
+}
+
+export function assertProvisioningNotCancelled(id: string): void {
+    if (provisioningCancelled(id)) throw new Error('Provisioning operation cancelled');
+}
+
+export function canCancelProvisioningJob(id: string): boolean {
+    return !!db().prepare(`SELECT 1 FROM server_deployments WHERE id=? AND server_deleted_at IS NULL
+        AND state IN ('queued','waiting','running','awaiting-health','interrupted')`).get(id);
+}
+
+/** Cancel remaining work, not effects already performed by the host. */
+export function cancelProvisioningJob(id: string, reason = 'Cancelled by operator'): boolean {
+    ensureHostSchema(db());
+    return db().transaction(() => {
+        if (provisioningCancelled(id)) return true;
+        if (!canCancelProvisioningJob(id)) return false;
+        const at = new Date().toISOString();
+        // Only commands never delivered can be withdrawn. Delivered commands retain
+        // their immutable IDs and receipts; replacements wait for their resolution.
+        const commands = db().prepare(`SELECT o.id FROM ec_host_operations o
+            WHERE o.state='pending' AND o.last_delivered_at IS NULL AND o.id IN (
+                SELECT operation_id FROM native_provisioning WHERE deployment_id=?
+                UNION SELECT stage_operation_id FROM native_provisioning WHERE deployment_id=?
+                UNION SELECT remove_operation_id FROM ec_host_migrations WHERE deployment_id=?)`).all(id, id, id) as { id: string }[];
+        for (const command of commands) {
+            db().prepare("UPDATE ec_host_operations SET state='failed',error=?,updated_at=? WHERE id=?").run(reason, at, command.id);
+            db().prepare("INSERT INTO ec_host_operation_events(operation_id,state,created_at,error) VALUES(?,'failed',?,?)").run(command.id, at, reason);
+        }
+        db().prepare(`UPDATE staged_file_changes SET state='cancelled' WHERE state='pending' AND EXISTS (
+            SELECT 1 FROM provisioning_sources s WHERE s.operation_id=? AND s.kind=staged_file_changes.source_kind
+                AND s.source_id=staged_file_changes.source_id AND s.server=staged_file_changes.server)`).run(id);
+        db().prepare('DELETE FROM restart_rollout WHERE operation_id=?').run(id);
+        db().prepare("UPDATE server_deployments SET state='cancelled',finished_at=?,stage=?,error=NULL WHERE id=?").run(at, reason, id);
+        recordProgress(id, { blocker: null, checks: [],
+            recovery: { state: 'released', at, operationId: id, message: 'Recovery capacity released by cancellation; this does not certify server health.' },
+            event: { kind: 'outcome', message: reason, details: { state: 'cancelled', finishedAt: at, error: null, warnings: [] } } });
+        return true;
+    })();
+}
+
+/** A cancelled controller job must not let a replacement overtake a delivered mutation. */
+export function unsettledCancelledCommand(server: string, replacementId: string): string | null {
+    const row = db().prepare(`SELECT d.id FROM server_deployments d
+        JOIN native_provisioning n ON n.deployment_id=d.id
+        LEFT JOIN ec_host_migrations m ON m.deployment_id=d.id
+        JOIN ec_host_operations o ON o.id=n.operation_id OR o.id=m.remove_operation_id
+        WHERE d.server=? AND d.server_deleted_at IS NULL AND d.state='cancelled'
+        AND (o.state NOT IN ('completed','failed')
+            OR (o.state='completed' AND o.id=n.operation_id AND n.files_json IS NOT NULL AND n.applied_at IS NULL))
+        AND NOT EXISTS (SELECT 1 FROM native_creation_cancellations c WHERE c.creation_id=d.id AND c.cleanup_id=?) LIMIT 1`).get(server, replacementId) as { id: string } | undefined;
+    return row?.id ?? null;
+}

src/lib/server/provisioning/history.ts

@@ -9,6 +9,7 @@ import { readProgress, recordProgress, type ProgressUpdate } from './progress.js
 import { bindNativeOperation } from './native.js';
 import { workerActivity } from '../pipeline/activity.js';
 import { canCancelCreation } from './cancel-creation.js';
+import { canCancelProvisioningJob, provisioningCancelled } from './cancellation.js';
 import { nativeDeployment } from '../backends/native.js';
 import { serverTitle } from '../../ui/server-labels.js';
 import { provisioningPageSize, provisioningPageOffset, type ProvisioningListOptions, type ProvisioningListPage, type ProvisioningListRow } from '../../provisioning-list.js';
@@ -121,7 +122,7 @@ export function getDeployment(id: string, options: { events?: 'all' | 'recent' }
             UNION ALL SELECT collection_id, deployment_id FROM server_collection_removals) m
             JOIN server_launch_collections c ON c.id = m.collection_id WHERE m.deployment_id = ? ORDER BY c.created_at, c.id LIMIT 1`).get(id) as { id: string; name: string } | undefined;
         return { deployment, backend: native ? 'native' as const : 'komodo' as const, agentOperation: agentOperation ?? null,
-            canCancelCreation: canCancelCreation(id), cancellation: cancellation ?? null,
+            canCancel: canCancelProvisioningJob(id), canCancelCreation: canCancelCreation(id), cancellation: cancellation ?? null,
             workerActivity: workerActivity(), events: deploymentEvents(id, options.events === 'recent'),
             activity: { ...activitySummary(id), failureExcerpt: latestFailureEvent(id) }, sources,
             recipe: launch?.recipe ?? null, blueprintName: launch?.blueprintName ?? null,
@@ -224,6 +225,9 @@ export function resumeVerification(id: string): boolean {
 }
 
 export function finishDeployment(id: string, result: { ok: boolean; message?: string; warnings?: string[] }): void {
+    // Creation failure cleanup deletes its own registry row, whose history trigger
+    // marks it cancelled. Preserve that real failure; explicit cancellation stays terminal.
+    if (provisioningCancelled(id) && !db().prepare("SELECT 1 FROM server_deployments WHERE id=? AND server_deleted_at IS NOT NULL AND error='Server deleted'").get(id)) return;
     const state = !result.ok ? 'failed' : result.warnings?.length ? 'warning' : 'done';
     db().transaction(() => {
         const finishedAt = new Date().toISOString();
@@ -257,7 +261,8 @@ const reviewDeployment = "d.state IN ('failed', 'interrupted', 'warning')";
 const deploymentListColumns = `d.id, d.server, d.server_deleted_at, d.action, d.state, d.stage,
         d.started_at, d.finished_at, d.triggered_by, d.error, d.warnings,
         (d.server_deleted_at IS NULL AND EXISTS(SELECT 1 FROM servers s WHERE s.name = d.server AND s.destroyed_at IS NULL)) AS server_exists,
-        (d.server_deleted_at IS NULL AND d.action = 'destroy' AND d.state = 'waiting' AND d.stage != ?) AS canCancelRemoval
+        (d.server_deleted_at IS NULL AND d.action = 'destroy' AND d.state = 'waiting' AND d.stage != ?) AS canCancelRemoval,
+        (d.server_deleted_at IS NULL AND d.state IN ('queued','waiting','running','awaiting-health','interrupted')) AS canCancel
     `;
 
 function deploymentListQuery(options: ProvisioningListOptions) {

src/lib/server/provisioning/host-drain.ts

@@ -1,4 +1,5 @@
 import { db } from '../db/index.js';
+import { provisioningCancelled } from './cancellation.js';
 import { nativeInstance, nativeRequest, type NativeInstance } from '../backends/native.js';
 import { getHost, getHostOperationStatus, listHosts, queueHostOperation, reserveHostPort, updateHost } from '../hosts/index.js';
 import { choosePlacement } from '../hosts/capacity.js';
@@ -43,6 +44,10 @@ export function reconcileHostDrains(): void {
             for (const instance of instances) {
                 const server = getServer(instance.server);
                 if (!server?.map_slug || activeBetaRemovalBlocker(instance.server)) continue;
+                // An explicit cancellation withdraws this instance's current drain attempt.
+                const latest = db().prepare(`SELECT d.state FROM ec_host_migrations m JOIN server_deployments d ON d.id=m.deployment_id
+                    WHERE m.instance_id=? AND d.server_deleted_at IS NULL ORDER BY d.rowid DESC LIMIT 1`).get(instance.id) as { state: string } | undefined;
+                if (latest?.state === 'cancelled') continue;
                 if (db().prepare(`SELECT 1 FROM server_deployments WHERE server=? AND server_deleted_at IS NULL
                     AND state IN ('queued','waiting','running','awaiting-health','interrupted')`).get(instance.server)) continue;
                 const recipe = { ...captureRecipePlan(server.game as 't5' | 't6', server.map_slug), image: instance.image,
@@ -67,7 +72,8 @@ export function reconcileHostDrains(): void {
 /** Reserve a destination, remove the source once, then hand creation to the normal health-verified rebuild queue. */
 export async function processMigrationRemoval(job: Job, instance: NativeInstance, migration: Migration,
     eligibility: () => string | null): Promise<void> {
-    const hold = (message: string) => deploymentStage(job.id, message, { blocker: { code: 'native-host', message, automatic: true } });
+    if (provisioningCancelled(job.id)) return;
+    const hold = (message: string) => { if (!provisioningCancelled(job.id)) deploymentStage(job.id, message, { blocker: { code: 'native-host', message, automatic: true } }); };
     if (migration.remove_operation_id) {
         const operation = getHostOperationStatus(migration.remove_operation_id);
         if (operation?.state === 'failed' && Date.now() - Date.parse(operation.updatedAt) >= 60_000) {
@@ -121,8 +127,10 @@ export async function processMigrationRemoval(job: Job, instance: NativeInstance
     // No source mutation is submitted unless another host can take this instance.
     choosePlacement(options);
     const health = await nativeRequest<{ state: string }>(instance, 'instance.inspect');
+    if (provisioningCancelled(job.id)) return;
     if (!['stopped','exited','created','dead','missing'].includes(health.state)) {
         const occupancy = await probePlayerEvidence(job.server);
+        if (provisioningCancelled(job.id)) return;
         if (occupancy.players !== 0) { hold(occupancy.players === null ? 'Waiting for a fresh local player count.' : `Waiting for ${occupancy.players} player(s) to leave.`); return; }
     }
     db().transaction(() => {

src/lib/server/provisioning/instances.ts

@@ -1,6 +1,7 @@
 import { listHosts } from '../hosts/index.js';
 import { fleetSettings } from '../settings.js';
 import { probePlayerEvidence, probeServerStatus } from '../fleet/status-evidence.js';
+import { assertProvisioningNotCancelled, provisioningCancelled } from './cancellation.js';
 import { beginDeployment, deploymentStage, finishDeployment, recordDeploymentEvent, REMOVING_SERVER, resumeCleanup, resumeVerification, claimRemoval } from './history.js';
 import { randomUUID } from 'node:crypto';
 import type { LaunchPreview } from '$lib/deployment-plan.js';
@@ -76,7 +77,7 @@ export function nextLobbyNumber(baseName: string, additionalNames: string[] = []
     const rows = db()
         .prepare(`SELECT name FROM servers WHERE name LIKE ? UNION
             SELECT j.server AS name FROM server_creation_jobs j JOIN server_deployments d ON d.id = j.deployment_id
-            WHERE d.server_deleted_at IS NULL AND (d.state IN ('queued', 'running', 'awaiting-health') OR (d.action = 'rebuild' AND (d.state = 'waiting' OR (d.state IN ('failed','interrupted') AND d.stage = 'Removing server files and registry'))))`)
+            WHERE d.server_deleted_at IS NULL AND (d.state IN ('queued', 'running', 'awaiting-health') OR (d.state = 'cancelled' AND d.submitted_at IS NOT NULL) OR (d.action = 'rebuild' AND (d.state = 'waiting' OR (d.state IN ('failed','interrupted') AND d.stage = 'Removing server files and registry'))))`)
         .all(`${baseName}_%`) as Array<{ name: string }>;
     const taken = new Set<number>();
     const prefix = `${baseName}_`;
@@ -100,7 +101,7 @@ export function nextAvailablePort(additionalPorts: number[] = [], releasingServe
     const rows = db()
         .prepare(`SELECT name, game_port FROM servers WHERE game_port IS NOT NULL UNION
             SELECT j.server AS name, j.port AS game_port FROM server_creation_jobs j JOIN server_deployments d ON d.id = j.deployment_id
-            WHERE d.server_deleted_at IS NULL AND (d.state IN ('queued', 'running', 'awaiting-health') OR (d.action = 'rebuild' AND (d.state = 'waiting' OR (d.state IN ('failed','interrupted') AND d.stage = 'Removing server files and registry'))))`)
+            WHERE d.server_deleted_at IS NULL AND (d.state IN ('queued', 'running', 'awaiting-health') OR (d.state = 'cancelled' AND d.submitted_at IS NOT NULL) OR (d.action = 'rebuild' AND (d.state = 'waiting' OR (d.state IN ('failed','interrupted') AND d.stage = 'Removing server files and registry'))))`)
         .all() as Array<{ name: string; game_port: number }>;
     const taken = new Set([...rows.filter(row => !releasingServers.includes(row.name)).map((r) => r.game_port), ...additionalPorts]);
     for (let p = config.portRange.min; p <= config.portRange.max; p++) {
@@ -159,9 +160,14 @@ function buildSpec(args: {
 }
 
 let creating = false;
-async function observeCommand<T>(operation: string, command: () => Promise<T>): Promise<T> {
-    try { return await command(); }
-    catch (err) {
+async function observeCommand<T>(operation: string, command: () => Promise<T>, retainReceipt = false): Promise<T> {
+    assertProvisioningNotCancelled(operation);
+    try {
+        const result = await command();
+        if (!retainReceipt) assertProvisioningNotCancelled(operation);
+        return result;
+    } catch (err) {
+        if (provisioningCancelled(operation)) throw err;
         recordProgress(operation, { command: { state: 'uncertain', message: 'No confirmed command result was received: ' + (err as Error).message } });
         throw err;
     }
@@ -314,6 +320,7 @@ export async function rebuildServer(id: string): Promise<void> {
     try {
         const result = job.stage === REMOVING_SERVER
             ? await cleanupDestroyedServer(job.server, id) : await destroyServerUnlocked(job.server, id);
+        if (provisioningCancelled(id)) return;
         if (!result.ok || result.warnings.length) {
             finishDeployment(id, { ok: false, message: result.ok ? 'Old server files could not be completely removed.' : result.message, warnings: result.warnings });
             return;
@@ -435,12 +442,16 @@ async function createServerUnlocked(input: CreateServerInput, prepared: Prepared
         return { ok: false, kind: 'fs', message: `failed to create ${hostDir}: ${(err as Error).message}` };
     }
 
+    assertProvisioningNotCancelled(operation);
     // Step 2: Komodo CreateDeployment.
     deploymentStage(operation, 'Creating Komodo deployment', { phase: 'create-deployment',
         checkpoint: { key: 'prepare', label: 'Server files prepared' },
         command: { state: 'attempted', message: 'Komodo deployment creation attempted.' } });
     const spec = buildSpec({ game, name, port, hostDir, plutoKey, recipe });
-    const createRes = await observeCommand(operation, () => komodoCreateDeployment(spec));
+    db().prepare('UPDATE server_deployments SET submitted_at=? WHERE id=?').run(new Date().toISOString(), operation);
+    // A confirmed creation must retain its identity for inspection/removal even if cancelled in flight.
+    const createRes = await observeCommand(operation, () => komodoCreateDeployment(spec), true);
+    if (!createRes.ok && provisioningCancelled(operation)) assertProvisioningNotCancelled(operation);
     if (!createRes.ok) {
         recordProgress(operation, { command: { state: createRes.kind === 'unreachable' || (createRes.httpStatus !== undefined && createRes.httpStatus >= 500) ? 'uncertain' : 'rejected', message: createRes.message } });
         await rm(hostDir, { recursive: true, force: true }).catch(() => {});
@@ -482,6 +493,7 @@ async function createServerUnlocked(input: CreateServerInput, prepared: Prepared
         return { ok: false, kind: 'db', message: `failed to insert server row: ${(err as Error).message}` };
     }
 
+    assertProvisioningNotCancelled(operation);
     // Step 4: apply Komodo tags so operators can filter the deployment list
     // by game/map. Best-effort — a tag failure doesn't roll back the server;
     // the deployment is fully functional without cosmetic tags. Fires in
@@ -718,6 +730,7 @@ async function destroyServerUnlocked(name: string, operation: string): Promise<D
     const pollDeadline = Date.now() + 30_000;
     while (Date.now() < pollDeadline) {
         const health = await komodoInspectHealthy(name);
+        assertProvisioningNotCancelled(operation);
         if (!health.ok) break;
         await new Promise((resolve) => setTimeout(resolve, 1500));
     }
@@ -740,6 +753,7 @@ async function destroyServerUnlocked(name: string, operation: string): Promise<D
 }
 
 async function cleanupDestroyedServer(name: string, operation: string): Promise<DestroyServerResult> {
+    assertProvisioningNotCancelled(operation);
     const blocker = activeBetaRemovalBlocker(name);
     if (blocker) return { ok: false, kind: 'conflict', message: blocker, warnings: [] };
     const row = getServer(name);
@@ -753,11 +767,13 @@ async function cleanupDestroyedServer(name: string, operation: string): Promise<
     const hostDir = row?.host_dir ?? serverDir(name);
     try {
         await rm(hostDir, { recursive: true, force: true });
+        assertProvisioningNotCancelled(operation);
         recordProgress(operation, { checkpoint: { key: 'files-removed', label: 'Instance files removed', phase: 'cleanup' } });
     } catch (err) {
         warnings.push(`rm ${hostDir} failed: ${(err as Error).message}`);
     }
 
+    assertProvisioningNotCancelled(operation);
     // The registry trigger preserves history and marks its old server identity.
     // Live inventory, rollback snapshots, and restart slots still cascade away.
     try {
@@ -802,6 +818,7 @@ async function provisionCreation(id: string): Promise<CreateServerResult> {
         }
         const result = await createServerUnlocked({ gameMode: job.game_mode, mapSlug: job.map_slug, createdBy: job.triggered_by },
             { ok: true, name: job.server, port: job.port, mapEntry, plutoKey }, id);
+        if (provisioningCancelled(id) && result.ok) return result;
         if (!result.ok) finishDeployment(id, result);
         else {
             db().transaction(() => {
@@ -812,6 +829,7 @@ async function provisionCreation(id: string): Promise<CreateServerResult> {
         }
         return result;
     } catch (err) {
+        if (provisioningCancelled(id)) throw err;
         const server = getServer(job.server);
         if (server?.game_port === job.port) {
             // A lost response may have happened after Komodo started the server.

src/lib/server/provisioning/native.ts

@@ -4,6 +4,8 @@ import { choosePlacement, availableHostPort, type PlacementOptions } from '../ho
 import { randomBytes, randomUUID } from 'node:crypto';
 import { posix } from 'node:path';
 import { db } from '../db/index.js';
+import { ensureHostSchema } from '../hosts/schema.js';
+import { unsettledCancelledCommand } from './cancellation.js';
 import { config, relDestForKind, serverDir } from '../config.js';
 import { nativeInstance, nativeRequest, unplacedNativeCreation, type NativeInstance } from '../backends/native.js';
 import { getHost, getHostOperationStatus, queueHostOperation, queueHostOperationSerialized, releaseHostPorts, reserveHostPort } from '../hosts/index.js';
@@ -244,7 +246,64 @@ function applyReceipt(job: Job, result: Record<string, unknown>) {
     })();
 }
 
-function hold(job: Job, message: string) { deploymentStage(job.id, message, { blocker: { code: 'native-host', message, automatic: true } }); }
+/** Retain confirmed file snapshots and instance identity even after observation is cancelled. */
+export function reconcileNativeProvisioningReceipt(id: string): void {
+    const job = db().prepare(`SELECT d.*,n.instance_id,n.generation,n.operation_id,n.files_json,n.applied_at
+        FROM server_deployments d JOIN native_provisioning n ON n.deployment_id=d.id
+        WHERE d.id=? AND d.server_deleted_at IS NULL`).get(id) as Job | undefined;
+    if (!job) return;
+    const migration = hostMigration(id);
+    if (job.state === 'cancelled' && migration?.remove_operation_id && !migration.removed_at) {
+        const removal = getHostOperationStatus(migration.remove_operation_id);
+        if (removal?.state === 'failed') {
+            db().prepare('DELETE FROM ec_host_ports WHERE host_id=? AND instance_id=?').run(migration.target_host_id, migration.instance_id);
+        } else if (removal?.state === 'completed') {
+            const original = nativeInstance(job.server);
+            if (original?.id === migration.instance_id && original.host_id === migration.source_host_id && original.generation === job.generation) {
+                db().prepare('UPDATE native_instances SET retired_at=? WHERE id=?').run(removal.updatedAt, original.id);
+                releaseHostPorts(original.id);
+                db().prepare('DELETE FROM servers WHERE name=?').run(job.server);
+                db().prepare('UPDATE ec_host_migrations SET removed_at=? WHERE deployment_id=?').run(removal.updatedAt, id);
+                recordProgress(id, { event: { kind: 'cleanup', message: 'Cancelled migration: source removal completed; replacement creation was not started.' } });
+            }
+        }
+    }
+    if (!job.operation_id) return;
+    const operation = getHostOperationStatus(job.operation_id);
+    if (operation?.state === 'failed') {
+        for (const file of JSON.parse(job.files_json ?? '[]') as FilePlan[]) {
+            db().prepare("UPDATE staged_file_changes SET state=? WHERE id=? AND state='applying'")
+                .run(job.state === 'cancelled' ? 'cancelled' : 'pending', file.id);
+        }
+        return;
+    }
+    if (operation?.state !== 'completed') return;
+    const instance = nativeInstance(job.server);
+    if (!instance || instance.id !== job.instance_id) return;
+    if (job.action === 'destroy') {
+        if (instance.generation !== job.generation) return;
+        db().prepare('UPDATE native_instances SET retired_at=? WHERE id=?').run(operation.updatedAt, instance.id);
+        releaseHostPorts(instance.id);
+        db().prepare('DELETE FROM servers WHERE name=?').run(job.server);
+        finishCreationCancellation(job.id);
+        return;
+    }
+    if (job.action === 'rebuild' && !hostMigration(id) && instance.generation === job.generation) {
+        db().prepare('UPDATE native_instances SET generation=generation+1 WHERE id=?').run(instance.id);
+        // Replacements queued while the delivered rebuild was settling bound the
+        // previous generation. Only untouched, newer jobs may follow this receipt.
+        if (job.state === 'cancelled') db().prepare(`UPDATE native_provisioning SET generation=?
+            WHERE instance_id=? AND generation=? AND operation_id IS NULL AND stage_operation_id IS NULL
+            AND deployment_id IN (SELECT newer.id FROM server_deployments newer
+                WHERE newer.rowid>(SELECT rowid FROM server_deployments WHERE id=?)
+                AND newer.server_deleted_at IS NULL AND newer.state IN ('queued','waiting'))`)
+            .run(job.generation + 1, instance.id, job.generation, id);
+    } else if (instance.generation !== job.generation && !(job.action === 'rebuild' && instance.generation === job.generation + 1)) return;
+    applyReceipt(job, (operation.result ?? {}) as Record<string, unknown>);
+    db().prepare('UPDATE native_provisioning SET applied_at=COALESCE(applied_at,?) WHERE deployment_id=?').run(operation.updatedAt, id);
+}
+
+function hold(job: Job, message: string) { if (!stillActive(job)) return; deploymentStage(job.id, message, { blocker: { code: 'native-host', message, automatic: true } }); }
 function stillActive(job: Job): boolean {
     return !!db().prepare(`SELECT 1 FROM server_deployments WHERE id=? AND server_deleted_at IS NULL
         AND state IN ('queued','waiting','running','awaiting-health','interrupted')`).get(job.id);
@@ -259,6 +318,26 @@ function complete(job: Job) {
 
 export async function processNativeProvisioning(eligibility: (operation: { id: string; server: string; action: string; state: string; stage: string }) => string | null,
     options: { wait?: boolean } = {}): Promise<void> {
+    ensureHostSchema(db());
+    const cancelled = db().prepare(`SELECT d.id FROM server_deployments d
+        JOIN native_provisioning n ON n.deployment_id=d.id
+        LEFT JOIN ec_host_operations o ON o.id=n.operation_id
+        LEFT JOIN ec_host_migrations m ON m.deployment_id=d.id
+        LEFT JOIN ec_host_operations removal ON removal.id=m.remove_operation_id
+        WHERE d.state='cancelled' AND d.server_deleted_at IS NULL AND (
+            (o.state='completed' AND (n.applied_at IS NULL OR d.action='destroy'))
+            OR (o.state='failed' AND EXISTS (SELECT 1 FROM staged_file_changes f JOIN provisioning_sources s
+                ON s.kind=f.source_kind AND s.source_id=f.source_id AND s.server=f.server
+                WHERE s.operation_id=d.id AND f.state='applying'))
+            OR (m.removed_at IS NULL AND (removal.state='completed' OR (removal.state='failed'
+                AND EXISTS (SELECT 1 FROM ec_host_ports p WHERE p.host_id=m.target_host_id AND p.instance_id=m.instance_id))))
+        )`).all() as { id: string }[];
+    for (const row of cancelled) {
+        try { reconcileNativeProvisioningReceipt(row.id); }
+        catch (error) {
+            deploymentStage(row.id, 'Cancelled; waiting to reconcile the delivered command: ' + (error as Error).message);
+        }
+    }
     reconcileHostDrains();
     const jobs = db().prepare(`SELECT d.*,n.instance_id,n.generation,n.stage_operation_id,n.operation_id,n.files_json,n.applied_at,q.selection
         FROM server_deployments d LEFT JOIN native_provisioning n ON n.deployment_id=d.id
@@ -346,6 +425,8 @@ async function placeQueuedCreation(job: Job, instance: NativeInstance | null): P
 
 async function processJob(job: Job, eligibility: (job: Job) => string | null) {
     if (!stillActive(job)) return;
+    const unsettled = unsettledCancelledCommand(job.server, job.id);
+    if (unsettled) { hold(job, 'Waiting for the already-delivered command from cancelled operation ' + unsettled + ' to settle.'); return; }
     let instance = nativeInstance(job.server);
     if (job.action === 'create' && !job.operation_id && !job.stage_operation_id && !job.submitted_at) {
         // A resumed bound job owns its original identity. An unplaced intent
@@ -514,6 +595,7 @@ async function processJob(job: Job, eligibility: (job: Job) => string | null) {
         bindAddress: instance.bind_address, excludeInstanceId: instance.id, thorough: false, dispatch: true });
     if (job.action === 'create' || job.action === 'rebuild') assertReadyLibrary(instance.host_id, instance.library_id);
     const checkOccupancy = async (): Promise<boolean> => {
+        if (!stillActive(job)) return false;
         // Fast controller check keeps occupied lobbies out of the operation slots.
         // The agent repeats this immediately before its local mutation.
         deploymentStage(job.id, 'Checking container and player occupancy before submission', { phase: 'eligibility', blocker: null });

src/lib/server/provisioning/restarts.ts

@@ -1,5 +1,6 @@
 import { fleetSettings } from '../settings.js';
 import { db } from '../db/index.js';
+import { provisioningCancelled } from './cancellation.js';
 import { komodoInspectHealthy, komodoRestart, komodoRuntimeAction } from '../integrations/komodo.js';
 import { probeServerStatus } from '../fleet/status-evidence.js';
 import { deploymentStage, finishDeployment } from './history.js';
@@ -45,6 +46,7 @@ export function rolloutWaitReason(server: string, operationId = '', action: Runt
 }
 
 export function rolloutBlocker(server: string, operationId = '', action: RuntimeAction = 'restart'): ProvisioningBlocker | null {
+    if (requestsInFlight.has(server)) return { code: 'operation-conflict', message: 'Waiting for the previous command submission to finish', automatic: true };
     const conflict = db().prepare(`SELECT id FROM server_deployments WHERE server = ? AND id != ? AND server_deleted_at IS NULL
         AND ((action IN ('start','stop','rebuild') AND state IN ('running','awaiting-health'))
             OR (action = 'destroy' AND state IN ('waiting','running')))`).get(server, operationId) as { id: string } | undefined;
@@ -74,6 +76,7 @@ export type RolloutRestartResult =
 export async function restartInRollout(server: string, options: { operationId: string; action: 'start' | 'restart' }): Promise<RolloutRestartResult> {
     const operationId = options.operationId;
     const waitReason = db().transaction(() => {
+        if (provisioningCancelled(operationId)) return 'Operation was cancelled';
         if (!db().prepare('SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL').get(operationId)) return 'Original server was deleted';
         const reason = rolloutWaitReason(server, operationId, options.action);
         if (reason) return reason;
@@ -111,6 +114,7 @@ export async function restartInRollout(server: string, options: { operationId: s
         game_started_at = NULL, game_ended_at = NULL WHERE name = ?`).run(server);
     try {
         const result = options.action === 'start' ? await komodoRuntimeAction(server, 'start') : await komodoRestart(server);
+        if (provisioningCancelled(operationId)) return { kind: 'failed', message: 'Operation was cancelled' };
         if (!db().prepare('SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL').get(operationId)) return { kind: 'failed', message: 'Original server was deleted' };
         if (result.ok) {
             db().transaction(() => {
@@ -138,6 +142,7 @@ export async function restartInRollout(server: string, options: { operationId: s
         })();
         return { kind: 'failed', message: `komodo: ${result.message}` };
     } catch (err) {
+        if (provisioningCancelled(operationId)) return { kind: 'failed', message: 'Operation was cancelled' };
         if (!db().prepare('SELECT 1 FROM server_deployments WHERE id = ? AND server_deleted_at IS NULL').get(operationId)) return { kind: 'failed', message: 'Original server was deleted' };
         const message = `komodo: ${(err as Error).message}`;
         db().transaction(() => {

src/lib/server/provisioning/worker.ts

@@ -1,6 +1,7 @@
 import { maintenanceAllowed } from '../maintenance/policies.js';
 import { resolveRecoveredHealthOperation } from '../maintenance/recovery.js';
 import { db } from '../db/index.js';
+import { provisioningCancelled } from './cancellation.js';
 import { komodoConfigured, rconConfigured } from '../config.js';
 import { getServer } from '../fleet/servers.js';
 import { deploymentStage, finishDeployment, REMOVING_SERVER } from './history.js';
@@ -137,6 +138,7 @@ async function run(): Promise<void> {
         if (nativeDeployment(operation.id)) continue;
         try { await processOperation(operation); }
         catch (err) {
+            if (provisioningCancelled(operation.id)) continue;
             // Observation failures are retried; remote mutations are never replayed.
             deploymentStage(operation.id, 'Waiting for verification: ' + (err as Error).message, {
                 checks: [], blocker: { code: 'observation-unavailable', message: (err as Error).message, automatic: true }
@@ -147,6 +149,7 @@ async function run(): Promise<void> {
 
 async function processOperation(operation: Operation): Promise<void> {
     const { id, server, action } = operation;
+    if (provisioningCancelled(id)) return;
     // An explicit cleanup resume follows confirmed remote deletion. Complete
     // that durable checkpoint even if the incident now needs attention or the
     // Watchdog is paused; never strand the reserved replacement identity.
@@ -194,6 +197,7 @@ async function processOperation(operation: Operation): Promise<void> {
         WHERE s.operation_id = ? AND s.kind = 'watchdog' AND i.kind = 'health' AND d.triggered_by LIKE 'watchdog:%'`).get(id);
     const epoch = latestGameSubmission(server);
     const health = await komodoInspectHealthy(server);
+    if (provisioningCancelled(id)) return;
     if (healthRecovery && epoch !== latestGameSubmission(server)) return;
     if (healthRecovery && !health.ok && !['stopped','exited','created','dead','unhealthy','starting','restarting'].includes(health.state?.toLowerCase() ?? '')) {
         deploymentStage(id, 'Waiting for current container health before automatic recovery', { phase: 'eligibility', checks: [],
@@ -235,6 +239,7 @@ async function processOperation(operation: Operation): Promise<void> {
     if (!claim.changes) return;
     if (action === 'start' || action === 'restart') {
         const result = await restartInRollout(server, { operationId: id, action });
+        if (provisioningCancelled(id)) return;
         if (result.kind === 'queued') {
             db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id);
             deploymentStage(id, result.message);
@@ -249,6 +254,7 @@ async function processOperation(operation: Operation): Promise<void> {
     // Persist intent first; a lost reply must not issue another stop later.
     try {
         const result = await komodoRuntimeAction(server, 'stop');
+        if (provisioningCancelled(id)) return;
         if (!result.ok) {
             recordProgress(id, { command: { state: result.kind === 'no-name' || result.kind === 'unconfigured' ? 'rejected' : 'uncertain', message: result.message } });
             finishDeployment(id, { ok: false, message: result.message }); return;
@@ -260,6 +266,7 @@ async function processOperation(operation: Operation): Promise<void> {
                 command: { state: 'accepted', message: 'Komodo accepted the stop request; stopped state still requires verification.' } });
         })();
     } catch (err) {
+        if (provisioningCancelled(id)) return;
         recordProgress(id, { command: { state: 'uncertain', message: (err as Error).message } });
         finishDeployment(id, { ok: false, message: (err as Error).message });
     }

src/lib/ui/provisioning-presentation.ts

@@ -163,6 +163,7 @@ export function isProvisioningEventPage(value: unknown): value is ProvisioningEv
 /** Validate every nested shape consumed by the page before replacing its last good snapshot. */
 export function isProvisioningDetail(value: unknown, id: string): value is ProvisioningDetail {
     if (!record(value) || !record(value.deployment)) return false;
+    if (value.canCancel !== undefined && typeof value.canCancel !== 'boolean') return false;
     if (value.canCancelCreation !== undefined && typeof value.canCancelCreation !== 'boolean') return false;
     if (value.cancellation !== undefined && value.cancellation !== null && !(record(value.cancellation)
         && text(value.cancellation.cleanupId) && text(value.cancellation.state) && text(value.cancellation.stage)

src/lib/ui/provisioning-refresh.ts

@@ -1,4 +1,4 @@
-export type ProvisioningMutation = 'resume' | 'cancel';
+export type ProvisioningMutation = 'resume' | 'cancel' | 'cancel-creation';
 
 export interface ProvisioningRefreshState<T> {
     id: string;

src/routes/api/deployments/[id]/cancel/+server.ts

@@ -1,14 +1,9 @@
 import { json } from '@sveltejs/kit';
-import { cancelRemoval } from '$lib/server/provisioning/history.js';
-import { cancelCreation } from '$lib/server/provisioning/cancel-creation.js';
+import { cancelProvisioningJob } from '$lib/server/provisioning/cancellation.js';
 import { kickProvisioning } from '$lib/server/pipeline/worker.js';
 
 export function POST({ params }: { params: { id: string } }) {
-    const cleanupId = cancelCreation(params.id);
-    if (cleanupId) {
+    if (!cancelProvisioningJob(params.id)) return json({ error: 'This operation can no longer be cancelled. Refresh to see its current state.' }, { status: 409 });
     void kickProvisioning();
-        return json({ ok: true, cleanupId });
-    }
-    if (!cancelRemoval(params.id)) return json({ error: 'This operation can no longer be cancelled. Refresh to see its current state.' }, { status: 409 });
     return json({ ok: true });
 }

src/routes/api/provisioning/[id]/cancel-creation/+server.ts

@@ -0,0 +1,10 @@
+import { json } from '@sveltejs/kit';
+import { cancelCreation } from '$lib/server/provisioning/cancel-creation.js';
+import { kickProvisioning } from '$lib/server/pipeline/worker.js';
+
+export function POST({ params }: { params: { id: string } }) {
+    const cleanupId = cancelCreation(params.id);
+    if (!cleanupId) return json({ error: 'This creation can no longer be removed. Refresh its current state.' }, { status: 409 });
+    void kickProvisioning();
+    return json({ ok: true, cleanupId });
+}

src/routes/provisioning/+page.svelte

@@ -60,7 +60,7 @@
             try { const response = await fetch(`/api/provisioning/${encodeURIComponent(id)}/cancel`, { method: 'POST' }); if (response.ok) cancelled++; else unavailable.push(id); }
             catch { unavailable.push(id); }
         }
-        cancellationNotice = `${cancelled} queued removal${cancelled === 1 ? '' : 's'} cancelled.${cancelled ? ' The servers covered by those cancelled requests and their files were kept.' : ''}${unavailable.length ? ` ${unavailable.length} could not be cancelled; refresh their operations to see whether removal has begun.` : ''}`;
+        cancellationNotice = `${cancelled} operation${cancelled === 1 ? '' : 's'} cancelled.${unavailable.length ? ` ${unavailable.length} could not be cancelled; refresh to see their current state.` : ''}`;
         cancelling = false; await load();
     }
     const tabs = $derived.by(() => [{id:'active',label:'Active',badge:active.length},{id:'history',label:'History'}]);
@@ -122,7 +122,7 @@
     {#if error}<p role="alert" class="error">{error} <button onclick={() => void load()}>Retry</button></p>{/if}
     {#if tab === 'active'}
     {#if needsReview}<div class="ui-summary"><span>{needsReview} recorded operations need review</span><Button size="sm" onclick={() => { historyState = 'attention'; tab = 'history'; saveContext(true); }}>Review history</Button></div>{/if}
-    {#if active.some(d => d.canCancelRemoval)}<div class="ui-summary"><span>Waiting removals can be withdrawn until removal begins. Running and completed removals cannot be reversed.</span><Button disabled={cancelling} onclick={() => cancelRequests(active.filter(d => d.canCancelRemoval).map(d => d.id))}>Cancel all waiting removals</Button></div>{/if}
+    {#if active.some(d => d.canCancel)}<div class="ui-summary"><span>Cancelling stops remaining work and health checks. Changes already made are not undone.</span><Button disabled={cancelling} onclick={() => cancelRequests(active.filter(d => d.canCancel).map(d => d.id))}>Cancel all active operations</Button></div>{/if}
     <section>
         <h2>In flight <span>{active.length}</span></h2>
         {#if loading}<p>Loading provisioning activity...</p>
@@ -165,7 +165,7 @@
         <div class="meta"><span>Started {fmt(d.started_at)}</span>{#if d.finished_at}<span>Finished {fmt(d.finished_at)}</span>{/if}<span>By {d.triggered_by}</span></div>
         {#if d.error}<p class="error">{d.error}</p>{/if}
         {#each warnings(d) as warning}<p class="error">{warning}</p>{/each}
-    </a>{#if d.canCancelRemoval && !d.server_deleted_at}<div class="row-actions"><Button size="sm" disabled={cancelling} onclick={() => cancelRequests([d.id])}>Cancel queued removal</Button></div>{/if}</article>
+    </a>{#if d.canCancel && !d.server_deleted_at}<div class="row-actions"><Button size="sm" disabled={cancelling} onclick={() => cancelRequests([d.id])}>Cancel operation</Button></div>{/if}</article>
 {/snippet}
 
 <style>

src/routes/provisioning/[id]/+page.svelte

@@ -89,13 +89,13 @@
             {#if detail.activity.failureExcerpt && detail.activity.failureExcerpt.message !== deployment.error && ['failed', 'interrupted'].includes(deployment.state)}<p class="failure-excerpt"><span>Last failure event · {fmt(detail.activity.failureExcerpt.created_at)}</span>{detail.activity.failureExcerpt.message}</p>{/if}
             {#if detail.warningMessages.length}<div class="warnings"><h3>Recorded warnings</h3><ul>{#each detail.warningMessages as warning}<li>{warning}</li>{/each}</ul></div>{/if}
 
-            {#if detail.canResumeVerification || detail.canResumeCleanup || detail.canResumeRebuild || detail.canCancelRemoval || detail.canCancelCreation}
+            {#if detail.canResumeVerification || detail.canResumeCleanup || detail.canResumeRebuild || detail.canCancel || detail.canCancelCreation}
                 <div class="recovery-actions">
                     {#if detail.canResumeVerification}<Button variant="primary" loading={view.busy === 'resume'} disabled={!!view.busy} onclick={() => controller.act('resume', 'Health verification requested. The existing instance will be observed without another creation or restart.')}>Resume health verification</Button>{/if}
                     {#if detail.canResumeRebuild}<Button variant="primary" loading={view.busy === 'resume'} disabled={!!view.busy} onclick={() => controller.act('resume', 'Rebuild cleanup requested. Fresh creation waits for local cleanup to finish.')}>Resume rebuild cleanup</Button>{/if}
                     {#if detail.canResumeCleanup}<Button variant="primary" loading={view.busy === 'resume'} disabled={!!view.busy} onclick={() => controller.act('resume', 'Local cleanup requested. Remote container deletion will not be repeated.')}>Resume cleanup</Button>{/if}
-                    {#if detail.canCancelRemoval}<div><Button disabled={!!view.busy} loading={view.busy === 'cancel'} onclick={() => controller.act('cancel', 'Queued removal cancelled. The server and its files were kept.')}>Cancel queued removal</Button><p class="action-help">Available until removal begins. Cancelling keeps the server and its files.</p></div>{/if}
-                    {#if detail.canCancelCreation}<div><Button variant="danger" disabled={!!view.busy} loading={view.busy === 'cancel'} onclick={() => controller.act('cancel', 'Creation cancelled. Follow cleanup below until removal is confirmed.')}>Cancel creation and remove server</Button><p class="action-help">Stops this launch and deletes its container and instance files, even if player status is unknown or players have joined. Shared game files and the saved server key are kept.</p></div>{/if}
+                    {#if detail.canCancel}<div><Button disabled={!!view.busy} loading={view.busy === 'cancel'} onclick={() => controller.act('cancel', 'Operation cancelled. Already-submitted commands may finish; no further steps will be started.')}>Cancel operation</Button><p class="action-help">Stops remaining work and health checks. Changes already made are not undone.</p></div>{/if}
+                    {#if detail.canCancelCreation}<div><Button variant="danger" disabled={!!view.busy} loading={view.busy === 'cancel-creation'} onclick={() => controller.act('cancel-creation', 'Creation cancelled. Follow cleanup below until removal is confirmed.')}>Cancel creation and remove server</Button><p class="action-help">Stops this launch and deletes its container and instance files, even if player status is unknown or players have joined. Shared game files and the saved server key are kept.</p></div>{/if}
                 </div>
             {/if}
             {#if detail.cancellation}<div class="recovery-note"><strong>{detail.cancellation.state === 'done' ? 'Cancellation cleanup completed' : 'Cancellation cleanup'}</strong><p>{detail.cancellation.error ?? detail.cancellation.stage}</p><a href={`/provisioning/${encodeURIComponent(detail.cancellation.cleanupId)}`}>View cleanup progress →</a></div>{/if}

tests/unit/beta-plans.test.ts

@@ -175,7 +175,8 @@ describe('durable beta test plans', () => {
         const { getBeta } = await import('../../src/lib/server/fleet/beta.js');
         const { db } = await import('../../src/lib/server/db/index.js');
         expect(getBeta(id)?.state).toBe('rolled-back');
-        expect(await readFile(scriptPath(names[0]), 'utf8')).toBe('main() {}');
+        // Rollback now replaces the unhealthy restart in the same worker pass.
+        expect(await readFile(scriptPath(names[0]), 'utf8')).toBe('original');
         expect(existsSync(scriptPath(names[1]))).toBe(false);
         expect(db().prepare("SELECT server, restart_state FROM deployment_servers").all()).toEqual([{ server: names[0], restart_state: 'pending' }]);
         const { sweep } = await import('../helpers/provisioning.js');

tests/unit/create-server.test.ts

@@ -20,7 +20,7 @@ vi.mock('../../src/lib/server/integrations/rcon.js', () => ({
     rconStatusResult: vi.fn(async () => ({ ok: true, endpoint: { host: 'localhost', port: 27016 }, rawReply: '',
         status: { players: 0, map: 'zombie_cosmodrome', gametype: null, roster: [] } }))
 }));
-vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}) }));
+vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}), kickProvisioning: vi.fn(async () => {}) }));
 
 let fixture: Fixture;
 const serverName = 't5_zombie_cosmodrome_1';
@@ -68,9 +68,9 @@ describe('queued removal cancellation', () => {
         resolveCount(0);
         await work;
         expect(komodoDestroyDeployment).not.toHaveBeenCalled();
-        expect(getDeployment(queued.deploymentId!)?.deployment).toMatchObject({ state: 'cancelled', server_exists: 1, stage: 'Removal cancelled' });
+        expect(getDeployment(queued.deploymentId!)?.deployment).toMatchObject({ state: 'cancelled', server_exists: 1, stage: 'Cancelled by operator' });
         expect(getDeployment(queued.deploymentId!)?.canCancelRemoval).toBe(false);
-        expect((await POST({ params: { id: queued.deploymentId! } })).status).toBe(409);
+        expect((await POST({ params: { id: queued.deploymentId! } })).status).toBe(200);
     });
 
     it('does not overwrite cancellation when the in-flight player check rejects', async () => {
@@ -760,3 +760,44 @@ describe('persistent server keys', () => {
             .toMatchObject({ ok: true, name: serverName });
     });
 });
+
+describe('cancelling submitted creation and destruction', () => {
+    it('retains a remotely created identity after cancellation without starting its container', async () => {
+        const { queueCreateServer, processCreations, nextAvailablePort, nextLobbyNumber } = await import('../../src/lib/server/provisioning/instances.js');
+        const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
+        const { getDeployment } = await import('../../src/lib/server/provisioning/history.js');
+        const { getServer } = await import('../../src/lib/server/fleet/servers.js');
+        const { komodoCreateDeployment, komodoDeploy } = await import('../../src/lib/server/integrations/komodo.js');
+        const queued = queueCreateServer(input); if (!queued.ok) throw new Error(queued.message);
+        let release!: (value: {ok:true}) => void;
+        vi.mocked(komodoCreateDeployment).mockImplementationOnce(() => new Promise(resolve => {release=resolve;}));
+        const work = processCreations();
+        await vi.waitFor(() => expect(release).toBeTypeOf('function'));
+        expect(cancelProvisioningJob(queued.deploymentId!)).toBe(true);
+        expect(nextAvailablePort()).not.toBe(queued.port);
+        expect(nextLobbyNumber('t5_zombie_cosmodrome')).toBe(2);
+        release({ok:true}); await work;
+        expect(getServer(queued.name)).toBeTruthy();
+        expect(komodoDeploy).not.toHaveBeenCalled();
+        expect(getDeployment(queued.deploymentId!)?.deployment.state).toBe('cancelled');
+    });
+
+    it('does not delete deployment metadata or files after cancelling container teardown', async () => {
+        const { createServer, destroyServer, processDestructions } = await import('../../src/lib/server/provisioning/instances.js');
+        const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
+        const { getDeployment } = await import('../../src/lib/server/provisioning/history.js');
+        const { getServer } = await import('../../src/lib/server/fleet/servers.js');
+        const { komodoDestroyDeployment, komodoDeleteDeployment } = await import('../../src/lib/server/integrations/komodo.js');
+        await createServer(input);
+        let release!: (value: {ok:true}) => void;
+        vi.mocked(komodoDestroyDeployment).mockImplementationOnce(() => new Promise(resolve => {release=resolve;}));
+        const queued=destroyServer(serverName); const work=processDestructions();
+        await vi.waitFor(() => expect(release).toBeTypeOf('function'));
+        expect(cancelProvisioningJob(queued.deploymentId!)).toBe(true);
+        release({ok:true}); await work;
+        expect(komodoDeleteDeployment).not.toHaveBeenCalled();
+        expect(getServer(serverName)).toBeTruthy();
+        expect(existsSync(join(fixture.t5,serverName))).toBe(true);
+        expect(getDeployment(queued.deploymentId!)?.deployment.state).toBe('cancelled');
+    });
+});

tests/unit/native-provisioning.test.ts

@@ -112,9 +112,9 @@ describe('native instance provisioning', () => {
         expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
     });
 
-    it('accepts cancellation through the provisioning API and rejects an already completed creation', async () => {
+    it('accepts creation cleanup through the provisioning API and rejects an already completed creation', async () => {
         const job = await queue();
-        const { POST } = await import('../../src/routes/api/provisioning/[id]/cancel/+server.js');
+        const { POST } = await import('../../src/routes/api/provisioning/[id]/cancel-creation/+server.js');
         const response = POST({ params: { id: job.deploymentId } });
         expect(response.status).toBe(200);
         expect(await response.json()).toMatchObject({ ok: true, cleanupId: expect.any(String) });
@@ -1173,3 +1173,151 @@ describe('automatic capacity admission and placement', () => {
         expect(db().prepare('SELECT port FROM native_instances').all()).toEqual([{ port: 28960 }, { port: 28960 }]);
     });
 });
+
+describe('native provisioning cancellation and rollback', () => {
+    it('cancels an unhealthy restart and prevents a late health response from reviving it', async () => {
+        const job = await queue(); await advance(); await acknowledge(job.deploymentId);
+        const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js');
+        const { POST } = await import('../../src/routes/api/provisioning/[id]/cancel/+server.js');
+        const { nativeRequest } = await import('../../src/lib/server/backends/native.js');
+        const { reportHostOperation } = await import('../../src/lib/server/hosts/index.js');
+        const restart = requestLifecycle(job.name, 'restart'); if (!restart.ok) throw new Error(restart.error);
+        await advance();
+        const command = (await operationFor(restart.deploymentId))!;
+        reportHostOperation(hostId, command.id, { state: 'completed' });
+        let release!: (value: never) => void;
+        vi.mocked(nativeRequest).mockImplementationOnce(() => new Promise(resolve => { release = resolve; }));
+        const pending = advance();
+        await vi.waitFor(() => expect(release).toBeTypeOf('function'));
+        expect(POST({ params: { id: restart.deploymentId } }).status).toBe(200);
+        release({ healthy: true, state: 'running' } as never);
+        await pending;
+        expect(await detail(restart.deploymentId)).toMatchObject({ state: 'cancelled' });
+        const replacement = requestLifecycle(job.name, 'restart'); if (!replacement.ok) throw new Error(replacement.error);
+        await advance();
+        expect(await operationFor(replacement.deploymentId)).toMatchObject({ kind: 'instance.restart', state: 'pending' });
+    });
+
+    it.each([false, true])('withdraws an undelivered restart or waits for a delivered one before replacement (delivered=%s)', async delivered => {
+        const job = await queue(); await advance(); await acknowledge(job.deploymentId);
+        const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js');
+        const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
+        const { pendingHostOperations, reportHostOperation } = await import('../../src/lib/server/hosts/index.js');
+        const first = requestLifecycle(job.name, 'restart'); if (!first.ok) throw new Error(first.error);
+        await advance(); const command = (await operationFor(first.deploymentId))!;
+        if (delivered) pendingHostOperations(hostId);
+        expect(cancelProvisioningJob(first.deploymentId)).toBe(true);
+        const next = requestLifecycle(job.name, 'restart'); if (!next.ok) throw new Error(next.error);
+        await advance();
+        if (delivered) {
+            expect(await operationFor(next.deploymentId)).toBeNull();
+            reportHostOperation(hostId, command.id, { state: 'completed' });
+            await advance();
+        } else {
+            expect((await operationFor(first.deploymentId))?.state).toBe('failed');
+            expect(pendingHostOperations(hostId).some(op => op.id === command.id)).toBe(false);
+        }
+        expect(await operationFor(next.deploymentId)).not.toBeNull();
+        expect((await detail(first.deploymentId)).state).toBe('cancelled');
+    });
+
+    it('rolls back a completed native activation even while its restart is stuck starting', async () => {
+        const job = await queue(); await advance(); await acknowledge(job.deploymentId);
+        const { db } = await import('../../src/lib/server/db/index.js');
+        const { createScript } = await import('../../src/lib/server/fleet/scripts.js');
+        const { requestRestart } = await import('../../src/lib/server/provisioning/requests.js');
+        const { queueFileChange } = await import('../../src/lib/server/fleet/staged-files.js');
+        const { reportHostOperation } = await import('../../src/lib/server/hosts/index.js');
+        const { rollbackBeta } = await import('../../src/lib/server/fleet/beta.js');
+        const { script, version } = createScript({ name: 'Broken beta', kind: 'script', targetRelPath: 'beta.gsc', content: Buffer.from('broken'), scopeType: 'game', scopeValue: 't5' });
+        const betaId = Number(db().prepare("INSERT INTO betas(script_id,script_version_id,trigger_kind,state) VALUES(?,?,'manual','running')").run(script.id,version.id).lastInsertRowid);
+        db().prepare('INSERT INTO beta_scripts(beta_id,script_id,script_version_id) VALUES(?,?,?)').run(betaId,script.id,version.id);
+        db().prepare("INSERT INTO beta_servers(beta_id,server,had_original,stage) VALUES(?,?,0,'awaiting-empty')").run(betaId,job.name);
+        db().prepare('INSERT INTO beta_server_scripts(beta_id,server,script_id,had_original) VALUES(?,?,?,0)').run(betaId,job.name,script.id);
+        queueFileChange({ sourceKind: 'beta', sourceId: String(betaId), server: job.name, kind: 'script', relPath: 'beta.gsc', content: Buffer.from('broken'), inventorySource: 'beta' });
+        const restart = requestRestart(job.name,{kind:'beta',id:String(betaId)});
+        await advance();
+        const staging = db().prepare('SELECT stage_operation_id FROM native_provisioning WHERE deployment_id=?').get(restart.id) as {stage_operation_id:string};
+        reportHostOperation(hostId,staging.stage_operation_id,{state:'completed'});
+        await advance();
+        const activation = (await operationFor(restart.id))!;
+        const files = JSON.parse((db().prepare('SELECT files_json FROM native_provisioning WHERE deployment_id=?').get(restart.id) as {files_json:string}).files_json);
+        remote.state = 'starting';
+        reportHostOperation(hostId,activation.id,{state:'completed',result:{before:files.map((f:{id:number;path:string})=>({...f,contentBase64:Buffer.from('working').toString('base64')}))}});
+        // Rollback must consume the receipt itself; the health worker has not caught up.
+        const rollback = await rollbackBeta(betaId);
+        expect((await detail(restart.id)).state).toBe('cancelled');
+        const replacement = db().prepare('SELECT provisioning_id FROM deployment_servers WHERE deployment_id=?').get(rollback.deploymentId) as {provisioning_id:string};
+        expect(replacement.provisioning_id).not.toBe(restart.id);
+        expect(db().prepare("SELECT content FROM staged_file_changes WHERE source_kind='campaign' AND source_id=?").get(rollback.deploymentId))
+            .toEqual({content:Buffer.from('working')});
+        await advance();
+        const stagedRollback = db().prepare('SELECT stage_operation_id FROM native_provisioning WHERE deployment_id=?').get(replacement.provisioning_id) as {stage_operation_id:string};
+        reportHostOperation(hostId,stagedRollback.stage_operation_id,{state:'completed'});
+        await advance();
+        expect(await operationFor(replacement.provisioning_id)).toMatchObject({state:'pending'});
+    });
+});
+
+describe('cancelled native mutation bookkeeping', () => {
+    it('keeps explicit cleanup available after cancelling an unsubmitted creation', async () => {
+        const job = await queue();
+        const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
+        const { cancelCreation, canCancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
+        expect(cancelProvisioningJob(job.deploymentId)).toBe(true);
+        expect(canCancelCreation(job.deploymentId)).toBe(true);
+        const cleanup = cancelCreation(job.deploymentId)!;
+        expect((await detail(cleanup)).state).toBe('done');
+    });
+
+    it.each(['rebuild','destroy'] as const)('reconciles a delivered %s receipt without resuming the cancelled job', async action => {
+        const job = await queue(); await advance(); await acknowledge(job.deploymentId);
+        const { db } = await import('../../src/lib/server/db/index.js');
+        const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js');
+        const { destroyServer } = await import('../../src/lib/server/provisioning/instances.js');
+        const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
+        const { pendingHostOperations, reportHostOperation } = await import('../../src/lib/server/hosts/index.js');
+        const result = action === 'rebuild' ? requestLifecycle(job.name,'rebuild') : destroyServer(job.name);
+        if (!result.ok || !result.deploymentId) throw new Error('Job was not queued');
+        const id = result.deploymentId;
+        await advance(); const operation = (await operationFor(id))!;
+        pendingHostOperations(hostId);
+        expect(cancelProvisioningJob(id)).toBe(true);
+        const replacement = action === 'rebuild' ? requestLifecycle(job.name, 'restart') : null;
+        if (replacement && !replacement.ok) throw new Error(replacement.error);
+        await advance();
+        reportHostOperation(hostId,operation.id,{state:'completed'});
+        await advance(); await advance();
+        if (replacement?.ok) expect(await operationFor(replacement.deploymentId)).toMatchObject({ kind: 'instance.restart', generation: 2 });
+        expect((await detail(id)).state).toBe('cancelled');
+        if(action === 'rebuild') expect(db().prepare('SELECT generation,retired_at FROM native_instances WHERE server=?').get(job.name)).toEqual({generation:2,retired_at:null});
+        else {
+            expect(db().prepare('SELECT * FROM ec_host_ports').all()).toEqual([]);
+            expect(db().prepare('SELECT * FROM servers WHERE name=?').get(job.name)).toBeUndefined();
+        }
+    });
+
+    it.each([false,true])('cancels a host migration without restarting it or leaking destination reservations (delivered=%s)', async delivered => {
+        const job = await queue(); await advance(); await acknowledge(job.deploymentId);
+        const hosts = await import('../../src/lib/server/hosts/index.js');
+        const { getLibraryVersion, updateHostLibraryCaches } = await import('../../src/lib/server/game-files/index.js');
+        const { db } = await import('../../src/lib/server/db/index.js');
+        const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
+        const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
+        const destination = hosts.createHost({name:'Destination',portMin:29000,portMax:29005});
+        hosts.enrollHost(hosts.createEnrollment(destination.id).token,inventory());
+        updateHostLibraryCaches(destination.id,[{libraryVersionId:libraryId,status:'ready',manifestSha256:getLibraryVersion(libraryId)!.manifestSha256}]);
+        hosts.updateHost(hostId,{mode:'draining'});
+        await processProvisioning();
+        const migration = db().prepare('SELECT deployment_id,remove_operation_id FROM ec_host_migrations').get() as {deployment_id:string;remove_operation_id:string};
+        expect(migration.remove_operation_id).toBeTruthy();
+        if(delivered) hosts.pendingHostOperations(hostId);
+        expect(cancelProvisioningJob(migration.deployment_id)).toBe(true);
+        if(delivered) hosts.reportHostOperation(hostId,migration.remove_operation_id,{state:'completed'});
+        await processProvisioning(); await processProvisioning();
+        expect((await detail(migration.deployment_id)).state).toBe('cancelled');
+        expect(db().prepare('SELECT * FROM ec_host_ports WHERE host_id=?').all(destination.id)).toEqual([]);
+        expect(db().prepare('SELECT * FROM ec_host_migrations').all()).toHaveLength(1);
+        expect(db().prepare('SELECT * FROM ec_host_ports WHERE host_id=?').all(hostId)).toHaveLength(delivered ? 0 : 2);
+    });
+});

tests/unit/provisioning-cancellation.test.ts

@@ -0,0 +1,57 @@
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+import { newFixture, type Fixture } from '../helpers/fixture.js';
+
+vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kickProvisioning: vi.fn(async () => {}) }));
+let fixture: Fixture;
+beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
+afterEach(async () => { await fixture.cleanup(); });
+
+describe('provisioning cancellation', () => {
+    it.each(['create', 'start', 'restart', 'rebuild', 'stop', 'destroy'])('cancels %s in every active phase and keeps cancellation terminal', async action => {
+        const { db } = await import('../../src/lib/server/db/index.js');
+        const { cancelProvisioningJob, canCancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
+        const { finishDeployment, getDeployment, listDeployments } = await import('../../src/lib/server/provisioning/history.js');
+        for (const state of ['queued', 'waiting', 'running', 'awaiting-health', 'interrupted']) {
+            const id = `${action}-${state}`;
+            db().prepare(`INSERT INTO server_deployments(id,server,action,state,stage,started_at,submitted_at,triggered_by)
+                VALUES(?,'server',?,?, 'In progress','2026-10-05',?,'test')`).run(id, action, state, state === 'waiting' ? null : '2026-10-05');
+            expect(canCancelProvisioningJob(id)).toBe(true);
+            expect(getDeployment(id)?.canCancel).toBe(true);
+            expect(listDeployments().find(row => row.id === id)?.canCancel).toBe(1);
+            expect(cancelProvisioningJob(id)).toBe(true);
+            expect(cancelProvisioningJob(id)).toBe(true);
+            finishDeployment(id, { ok: true });
+            expect(getDeployment(id)?.deployment).toMatchObject({ state: 'cancelled', stage: 'Cancelled by operator' });
+            expect(getDeployment(id)?.canCancel).toBe(false);
+            expect(db().prepare("SELECT COUNT(*) n FROM server_deployment_events WHERE deployment_id=? AND kind='outcome'").get(id)).toEqual({ n: 1 });
+        }
+    });
+
+    it('releases only the cancelled job’s recovery slot and cancels its pending file changes', async () => {
+        const { db } = await import('../../src/lib/server/db/index.js');
+        const { beginDeployment } = await import('../../src/lib/server/provisioning/history.js');
+        const { queueFileChange } = await import('../../src/lib/server/fleet/staged-files.js');
+        const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
+        db().prepare("INSERT INTO servers(name,komodo_name) VALUES('one','one'),('two','two')").run();
+        const id = beginDeployment('one','restart');
+        const other = beginDeployment('two','restart');
+        db().prepare("INSERT INTO restart_rollout(server,requested_at,operation_id) VALUES('one','2026-10-05',?),('two','2026-10-05',?)").run(id, other);
+        db().prepare("INSERT INTO provisioning_sources(operation_id,kind,source_id,server) VALUES(?,'campaign','source','one')").run(id);
+        queueFileChange({ sourceKind: 'campaign', sourceId: 'source', server: 'one', kind: 'script', relPath: 'test.gsc', content: Buffer.from('queued'), inventorySource: 'test' });
+        expect(cancelProvisioningJob(id)).toBe(true);
+        expect(db().prepare('SELECT operation_id FROM restart_rollout').all()).toEqual([{ operation_id: other }]);
+        expect(db().prepare('SELECT state FROM staged_file_changes').get()).toEqual({ state: 'cancelled' });
+    });
+
+    it('rejects missing, completed and deleted-server jobs through the cancellation API', async () => {
+        const { db } = await import('../../src/lib/server/db/index.js');
+        const { POST } = await import('../../src/routes/api/deployments/[id]/cancel/+server.js');
+        for (const state of ['done','warning','failed']) {
+            db().prepare(`INSERT INTO server_deployments(id,server,action,state,stage,started_at,triggered_by) VALUES(?,'server','restart',?,'Finished','2026-10-05','test')`).run(state,state);
+            expect(POST({ params: { id: state } }).status).toBe(409);
+        }
+        expect(POST({ params: { id: 'missing' } }).status).toBe(409);
+        db().prepare(`INSERT INTO server_deployments(id,server,action,state,stage,started_at,triggered_by,server_deleted_at) VALUES('deleted','server','restart','running','Running','2026-10-05','test','2026-10-05')`).run();
+        expect(POST({ params: { id: 'deleted' } }).status).toBe(409);
+    });
+});

tests/unit/provisioning.test.ts

@@ -535,3 +535,27 @@ describe('Watchdog provisioning and telemetry evidence', () => {
         expect(await operation(rebuild.deploymentId)).toMatchObject({ state: 'done' });
     });
 });
+
+describe('cancelling submitted legacy commands', () => {
+    it.each(['start', 'restart', 'stop'] as const)('ignores a late %s response after cancellation', async action => {
+        const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
+        const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
+        const { db } = await import('../../src/lib/server/db/index.js');
+        const komodo = await import('../../src/lib/server/integrations/komodo.js');
+        let release!: (value: { ok: true }) => void;
+        const command = action === 'restart' ? komodo.komodoRestart : komodo.komodoRuntimeAction;
+        vi.mocked(command).mockImplementationOnce(() => new Promise(resolve => { release = resolve; }));
+        const id = await queue(action);
+        const pending = processProvisioning();
+        await vi.waitFor(() => expect(release).toBeTypeOf('function'));
+        expect(cancelProvisioningJob(id)).toBe(true);
+        release({ ok: true });
+        await pending;
+        expect(await operation(id)).toMatchObject({ state: 'cancelled', stage: 'Cancelled by operator' });
+        expect(db().prepare('SELECT * FROM restart_rollout WHERE operation_id=?').all(id)).toEqual([]);
+        const replacement = await queue(action);
+        await processProvisioning();
+        expect(await operation(replacement)).toMatchObject({ state: 'awaiting-health' });
+        expect(command).toHaveBeenCalledTimes(2);
+    });
+});

tests/unit/staged-files.test.ts

@@ -395,7 +395,7 @@ describe('shared activation and partial writes', () => {
             .get('legacy-drift') as { provisioning_id: string; restart_state: string; error: string };
         const restoration = db().prepare('SELECT provisioning_id FROM deployment_servers WHERE deployment_id = ?')
             .get(rollback.deploymentId) as { provisioning_id: string };
-        expect(old).toMatchObject({ restart_state: 'failed', error: expect.stringContaining('changed while queued') });
+        expect(old).toMatchObject({ restart_state: 'failed', error: expect.stringContaining('Superseded by rollback') });
         expect(restoration.provisioning_id).not.toBe(old.provisioning_id);
     });
 
@@ -438,3 +438,26 @@ describe('shared activation and partial writes', () => {
         expect(komodoRestart).not.toHaveBeenCalled();
     });
 });
+
+describe('rollback supersedes provisioning', () => {
+    it.each(['when-empty', 'manual'] as const)('replaces an unhealthy beta restart with a %s rollback without waiting for recovery', async restart => {
+        const beta = await queueChange('beta');
+        const { db } = await import('../../src/lib/server/db/index.js');
+        const { rollbackBeta } = await import('../../src/lib/server/fleet/beta.js');
+        const { komodoInspectHealthy, komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
+        await tick('beta');
+        const old = db().prepare('SELECT provisioning_id FROM beta_servers WHERE beta_id=?').get(beta.sourceId) as { provisioning_id: string };
+        expect(db().prepare('SELECT state FROM server_deployments WHERE id=?').get(old.provisioning_id)).toEqual({ state: 'awaiting-health' });
+        vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'exited', reason: 'container failed to start' });
+        const rollback = await rollbackBeta(Number(beta.sourceId), { restart });
+        expect(db().prepare('SELECT state FROM server_deployments WHERE id=?').get(old.provisioning_id)).toEqual({ state: 'cancelled' });
+        expect(db().prepare('SELECT * FROM restart_rollout WHERE operation_id=?').all(old.provisioning_id)).toEqual([]);
+        const replacement = db().prepare('SELECT provisioning_id,restart_state FROM deployment_servers WHERE deployment_id=?').get(rollback.deploymentId) as { provisioning_id: string; restart_state: string };
+        expect(replacement.provisioning_id).not.toBe(old.provisioning_id);
+        await tick('campaign');
+        expect(await readFile(destination())).toEqual(restart === 'manual' ? after : before);
+        expect(komodoRestart).toHaveBeenCalledTimes(restart === 'manual' ? 1 : 2);
+        expect(db().prepare('SELECT state FROM server_deployments WHERE id=?').get(replacement.provisioning_id))
+            .toEqual({ state: restart === 'manual' ? 'waiting' : 'awaiting-health' });
+    });
+});