Verify Plutonium revisions during provisioning

README.md

@@ -99,7 +99,8 @@ requests a rebuild with a linked Campaign. **Rebuild deletes all instance files.
 Plutonium updates are applied by Plutainer's startup updater
 (`PLUTAINER_AUTO_UPDATE` must be enabled). Check versions and choose **Update on
 restart**, or opt into automatic updates for verified outdated servers. Revision
-checks run hourly. Unknown or stale revisions remain unverified. A healthy restart
+checks run hourly, and Create and Rebuild verify and record the running revision
+before completing. Unknown or stale revisions remain unverified. A healthy restart
 can still fail update verification; Maintenance reports that outcome without
 rebuilding or repeatedly restarting the server. Electric Cherry does not download
 binaries or pin a runtime revision.
@@ -166,7 +167,7 @@ Slots persist in SQLite across app restarts, and overlapping timer/API triggers
 share the active run. Beta options to skip health or match gates still work;
 skipping a beta gate does not release the fleet's recovery slot early. New-server
 creation uses a separate shared startup queue, holding its slot through container,
-game-status, and selected-map verification.
+game-status, selected-map, and Plutonium revision verification.
 
 No token, no session, no auth freeze. If Komodo or RCON aren't configured,
 the sweep marks pending rows `manual` so nothing hangs forever.
@@ -207,9 +208,11 @@ page tracks both actions. **Fill missing servers** remains the default.
 All jobs and capacity reservations are committed together before provisioning;
 one collection page tracks each server's independent progress and failures.
 New servers start one at a time by default across all collections and individual
-launches. A startup slot stays occupied until Komodo is healthy and RCON or
-fresh telemetry reports the selected map; an unready server holds the queue until it recovers
-or is removed. The queue survives application restarts. The collection preview
+launches. A startup slot stays occupied until Komodo is healthy, RCON or fresh
+telemetry reports the selected map, and RCON verifies the current Plutonium
+production revision or newer. Unknown release or runtime revisions hold the queue
+while verification retries; a confirmed older runtime fails provisioning. The
+queue survives application restarts. The collection preview
 shows the startup limit, and its progress page shows active startups and each
 server's place in the shared queue. Adjust **Concurrent server startups** in
 **Settings → Fleet settings** to increase the limit. Each worker pass starts at most the
@@ -431,7 +434,7 @@ be restored by this upgrade. Existing history is marked deleted where the server
 is already absent; instances that reused a name before upgrading cannot always be
 distinguished.
 
-Provisioning (`/provisioning`, `/api/provisioning`) owns Create, Start, Restart, Rebuild, Stop, and Destroy: request persistence, eligibility and occupancy checks, execution, recovery, and history. Campaigns, betas, and scheduled maintenance request lifecycle work through Provisioning and link to the resulting operation. They retain their file, test-evidence, and version-verification workflows. Pending automated restarts appear in Provisioning before execution. History survives server removal. New creation requests return HTTP 202 with an operation ID and open the provisioning detail page. Creation is complete only when Komodo reports healthy and RCON or fresh telemetry verifies game status with an engine map that exactly matches the selected map. Queued requests reserve their server name and port; queued creation and pending health checks survive application restarts. The detail page records startup progress and explains any health or map mismatch that is still blocking completion.
+Provisioning (`/provisioning`, `/api/provisioning`) owns Create, Start, Restart, Rebuild, Stop, and Destroy: request persistence, eligibility and occupancy checks, execution, recovery, and history. Campaigns, betas, and scheduled maintenance request lifecycle work through Provisioning and link to the resulting operation. They retain their file, test-evidence, and version-verification workflows. Pending automated restarts appear in Provisioning before execution. History survives server removal. New creation requests return HTTP 202 with an operation ID and open the provisioning detail page. Create and Rebuild complete only when Komodo reports healthy, RCON or fresh telemetry verifies the selected engine map and game status, and RCON reports a Plutonium runtime revision at least as new as the current production release. Provisioning records the release and runtime evidence so new servers have verified version status immediately. Unknown release or runtime revisions are retried; a confirmed older runtime fails with a clear error and no automatic restart. Queued requests reserve their server name and port; queued creation and pending verification survive application restarts. The detail page records startup progress and explains any health, map, or revision issue blocking completion.
 
 Each server deployment opens a detail page with its outcome, blocker, permitted
 recovery actions, grouped execution phases, and timestamped verification evidence.

docs/provisioning.md

@@ -9,10 +9,10 @@ server, choose **Manage server** or open its **Server actions** menu.
 
 | Action | Behavior |
 | --- | --- |
-| Create | Create a new instance using the latest promoted files and verify container health, RCON, and the selected map. |
+| Create | Create a new instance using the latest promoted files and verify container health, game status, the selected map, and the Plutonium revision. |
 | Start | Start the existing container and verify container health and RCON. |
 | Restart | Wait until empty, restart the existing container, and verify container health and RCON. |
-| Rebuild | Wait until empty, destroy the instance, delete all server files and its registration, then create a fresh instance in the same server slot and port. Verify container health, RCON, and map. |
+| Rebuild | Wait until empty, destroy the instance, delete all server files and its registration, then create a fresh instance in the same server slot and port. Verify container health, game status, map, and Plutonium revision. |
 | Stop | Wait until empty, stop the container, and verify that Komodo reports it stopped. Keep the instance for a later Start. |
 | Destroy | Wait until empty, remove the container and Komodo deployment, then delete server files and registration. |
 
@@ -38,6 +38,16 @@ proceed until the old files are fully removed. An interrupted fresh startup can
 resume health verification, without repeating teardown or creation. Inspect other
 failures before making another request.
 
+Create and Rebuild finish only after RCON reports a runtime revision at least as
+new as the current Plutonium production release. The worker checks the production
+manifest and records the runtime revision for fleet and server version status as
+part of provisioning. An unavailable release or unknown runtime revision keeps the
+job awaiting verification and holds its startup slot while the worker retries.
+A confirmed older runtime fails the operation with its expected and observed
+revisions; it does not trigger an automatic restart. Check Plutainer's startup
+updater configuration and server logs before retrying. Telemetry can verify game
+status and map, but cannot verify the Plutonium revision.
+
 Start and Restart share the rolling restart limit with campaigns, betas, and
 maintenance. A successful API reply is followed by a recovery delay and actual
 container and game-status checks. Failed or ambiguous restart requests retain their recovery
@@ -86,7 +96,9 @@ collection. A completed provisioning job does not imply that its requester has
 finished its own verification.
 
 New operations record structured verification evidence alongside their activity:
-the check result, source, evaluation time, and expected/observed map when relevant.
+the check result, source, evaluation time, and expected/observed map or Plutonium
+revision when relevant. Create and Rebuild show **Verify Plutonium revision**, with
+separate evidence for the production manifest and the running revision from RCON.
 Game status can be verified by RCON or accepted fresh telemetry. Engine-map checks
 do not independently verify a T6 start location. Passed checks describe their
 recorded observation; later checks can fail or become unknown. Completion describes

src/lib/provisioning-detail.ts

@@ -3,13 +3,13 @@ import type { LaunchRecipeSummary } from './deployment-plan.js';
 
 export const provisioningPhases = ['requested', 'eligibility', 'startup-queue', 'prepare', 'create-deployment',
     'container-action', 'remove-container', 'delete-deployment', 'cleanup', 'verify-container', 'verify-game',
-    'verify-map', 'verify-stopped'] as const;
+    'verify-map', 'verify-plutonium', 'verify-stopped'] as const;
 export type ProvisioningPhase = typeof provisioningPhases[number];
 export type EvidenceResult = 'pass' | 'waiting' | 'fail' | 'unknown';
-export type EvidenceSource = 'komodo' | 'rcon' | 'telemetry' | 'combined' | null;
+export type EvidenceSource = 'komodo' | 'rcon' | 'telemetry' | 'combined' | 'manifest' | null;
 
 export interface ProvisioningCheck {
-    key: 'container' | 'game' | 'map' | 'occupancy' | 'stopped';
+    key: 'container' | 'game' | 'map' | 'plutonium-release' | 'plutonium-version' | 'occupancy' | 'stopped';
     label: string;
     result: EvidenceResult;
     source: EvidenceSource;

src/lib/server/integrations/plutonium.ts

@@ -35,20 +35,30 @@ export function runningVersions(): PlutoniumVersion[] {
     });
 }
 export async function probeVersion(server: string): Promise<number | null> {
+    const original = getServer(server);
+    if (!original) return null;
     const epoch = latestGameSubmission(server);
     const recovering = restartRecoveryPending(server);
-    const health = !recovering && getServer(server)?.desired_state !== 'stopped'
+    const health = !recovering && original.desired_state !== 'stopped'
         ? await komodoInspectHealthy(server).catch(() => ({ ok: false })) : { ok: false };
     let result = !health.ok ? { revision: null, error: 'Waiting for a healthy running server.' }
         : await rconPlutoniumVersion(server).catch(() => ({ revision: null, error: 'Version check failed.' }));
-    if (!getServer(server)) return null;
-    if (epoch !== latestGameSubmission(server) || restartRecoveryPending(server)) {
+    const current = getServer(server);
+    // An older probe must not clear evidence already recorded for a new process
+    // or replacement registration, including replacements with no submission yet.
+    if (!current || current.created_at !== original.created_at || epoch !== latestGameSubmission(server)) return null;
+    const checkedAt = nowIso();
+    if (restartRecoveryPending(server)) {
         result = { revision: null, error: 'Waiting for restart recovery.' };
+    } else if (current.desired_state === 'stopped') {
+        result = { revision: null, error: 'Server is intentionally stopped.' };
+    } else if (epoch && checkedAt <= epoch) {
+        result = { revision: null, error: 'Waiting for a version observation after startup.' };
     }
     // Clear the old revision on failure: it is no longer evidence of what is running.
     db().prepare(`INSERT INTO plutonium_versions (server, revision, checked_at, error) VALUES (?, ?, ?, ?)
         ON CONFLICT(server) DO UPDATE SET revision = excluded.revision, checked_at = excluded.checked_at, error = excluded.error`)
-        .run(server, result.revision, nowIso(), result.error);
+        .run(server, result.revision, checkedAt, result.error);
     return result.revision;
 }
 
@@ -59,6 +69,23 @@ export function checkPlutonium(): Promise<void> {
     return checking;
 }
 async function check(): Promise<void> {
+    await checkRelease();
+    const servers = listServers();
+    for (let i = 0; i < servers.length; i += 4) {
+        await Promise.all(servers.slice(i, i + 4).map(s => probeVersion(s.name)));
+    }
+    writeSetting('plutonium_versions_checked_at', nowIso());
+}
+
+let checkingRelease: Promise<void> | null = null;
+/** Refresh the published target without probing unrelated fleet servers. */
+export function checkRelease(): Promise<void> {
+    if (checkingRelease) return checkingRelease;
+    checkingRelease = fetchRelease().finally(() => { checkingRelease = null; });
+    return checkingRelease;
+}
+
+async function fetchRelease(): Promise<void> {
     const previous = release();
     try {
         const response = await fetch(PLUTONIUM_MANIFEST_URL, { signal: AbortSignal.timeout(config.httpTimeoutMs), redirect: 'error' });
@@ -69,8 +96,4 @@ async function check(): Promise<void> {
     } catch (err) {
         writeSetting('plutonium_release', { ...previous, checkedAt: nowIso(), error: (err as Error).message });
     }
-    const servers = listServers();
-    for (let i = 0; i < servers.length; i += 4) {
-        await Promise.all(servers.slice(i, i + 4).map(s => probeVersion(s.name)));
-    }
 }

src/lib/server/maintenance/runtime.ts

@@ -98,7 +98,11 @@ export function maintenanceTick(): Promise<void> {
 }
 async function runTick(): Promise<void> {
     const latest = release();
-    if (!latest.checkedAt || Date.now() - Date.parse(latest.checkedAt) >= HOUR) await checkPlutonium();
+    // Provisioning refreshes the manifest independently; that must not postpone
+    // the hourly version checks for the rest of the fleet.
+    const versionsCheckedAt = readSetting<string | null>('plutonium_versions_checked_at', null);
+    if (!latest.checkedAt || Date.now() - Date.parse(latest.checkedAt) >= HOUR
+        || !versionsCheckedAt || Date.now() - Date.parse(versionsCheckedAt) >= HOUR) await checkPlutonium();
     if (maintenanceAllowed('updates') && freshRelease() !== null) queueUpdates(undefined, true);
     db().transaction(() => {
         const plan = schedule();

src/lib/server/provisioning/instances.ts

@@ -27,6 +27,7 @@ import { recordScript } from '../fleet/inventory.js';
 import { getVersion } from '../fleet/scripts.js';
 import type { ProvisioningBlocker } from '../../provisioning-detail.js';
 import { recordProgress, type ProgressUpdate } from './progress.js';
+import { checkRelease, freshRelease, probeVersion, release } from '../integrations/plutonium.js';
 
 export type CreateServerFailure =
     | 'bad-request'
@@ -871,6 +872,7 @@ async function runCreations(): Promise<CreationOutcomes> {
             continue;
         }
         const checks: NonNullable<ProgressUpdate['checks']> = [];
+        let verifyingPlutonium = false;
         try {
             const komodo = await komodoInspectHealthy(job.server);
             if (!stillCheckingCreation(id)) continue;
@@ -893,21 +895,60 @@ async function runCreations(): Promise<CreationOutcomes> {
                     href: `/servers/${encodeURIComponent(job.server)}?tab=config`, label: 'Map configuration' } });
                 continue;
             }
+            verifyingPlutonium = true;
+            recordProgress(id, { phase: 'verify-plutonium', checks });
+            // Refresh only the release, then probe this instance. Fleet-wide checks
+            // can run independently and must not be required to complete a launch.
+            await checkRelease();
+            if (!stillCheckingCreation(id)) continue;
+            const revision = await probeVersion(job.server);
+            if (!stillCheckingCreation(id)) continue;
+            const target = freshRelease();
+            const published = release();
+            const version = db().prepare('SELECT checked_at, error FROM plutonium_versions WHERE server = ?').get(job.server) as
+                { checked_at: string; error: string | null } | undefined;
+            checks.push({ key: 'plutonium-release', label: 'Published Plutonium revision', source: 'manifest',
+                result: target === null ? 'unknown' : 'pass', checkedAt: published.checkedAt ?? undefined, actual: target,
+                message: target === null ? published.error ?? 'The production revision could not be verified.' : null });
+            checks.push({ key: 'plutonium-version', label: 'Running Plutonium revision', source: 'rcon',
+                result: revision === null || target === null ? 'unknown' : revision < target ? 'fail' : 'pass',
+                checkedAt: version?.checked_at, expected: target, actual: revision, message: version?.error });
+            if (target === null || revision === null) {
+                const message = target === null ? 'Waiting for the Plutonium production manifest: ' + (published.error ?? 'revision unavailable')
+                    : 'Waiting for the running Plutonium revision: ' + (version?.error ?? 'version unavailable');
+                deploymentStage(id, message, { phase: 'verify-plutonium', checks,
+                    blocker: { code: target === null ? 'plutonium-release' : 'plutonium-version', message, automatic: true } });
+                continue;
+            }
+            if (revision < target) {
+                const message = `Plutonium verification failed: running r${revision}, production is r${target}. Inspect the startup updater before retrying.`;
                 db().transaction(() => {
-                deploymentStage(id, 'Verified Komodo healthy, ' + (status.source === 'rcon' ? 'RCON healthy' : 'fresh telemetry') + ', and map ' + actualMap, {
-                    phase: 'verify-map', checks, blocker: null, checkpoints: [
+                    deploymentStage(id, message, { phase: 'verify-plutonium', checks, blocker: null });
+                    finishDeployment(id, { ok: false, message });
+                })();
+                continue;
+            }
+            db().transaction(() => {
+                deploymentStage(id, 'Verified Komodo healthy, ' + (status.source === 'rcon' ? 'RCON healthy' : 'fresh telemetry')
+                    + `, map ${actualMap}, and Plutonium r${revision} (production r${target})`, {
+                    phase: 'verify-plutonium', checks, blocker: null, checkpoints: [
                         { key: 'container-action', label: 'Container startup verified' },
                         { key: 'verify-container', label: 'Container readiness verified' },
                         { key: 'verify-game', label: 'Game status verified' },
-                        { key: 'verify-map', label: 'Selected engine map verified' }
+                        { key: 'verify-map', label: 'Selected engine map verified' },
+                        { key: 'verify-plutonium', label: `Plutonium r${revision} verified against production r${target}` }
                     ]
                 });
                 finishDeployment(id, { ok: true });
             })();
         } catch (err) {
             if (!stillCheckingCreation(id)) continue;
-            checks.push({ key: checks.length ? 'game' : 'container', label: checks.length ? 'Game status' : 'Container readiness', source: null, result: 'unknown', message: (err as Error).message });
-            deploymentStage(id, 'Waiting for health verification: ' + (err as Error).message, { phase: checks.length > 1 ? 'verify-game' : 'verify-container', checks,
+            const key = verifyingPlutonium ? 'plutonium-version' : checks.length ? 'game' : 'container';
+            const remaining = checks.filter(check => check.key !== key);
+            remaining.push({ key, label: verifyingPlutonium ? 'Running Plutonium revision' : checks.length ? 'Game status' : 'Container readiness',
+                source: null, result: 'unknown', message: (err as Error).message });
+            deploymentStage(id, 'Waiting for verification: ' + (err as Error).message, {
+                phase: verifyingPlutonium ? 'verify-plutonium' : checks.length ? 'verify-game' : 'verify-container', checks: remaining,
                 blocker: { code: 'observation-unavailable', message: (err as Error).message, automatic: true } });
         }
     }

src/lib/server/provisioning/progress.ts

@@ -34,9 +34,9 @@ export function readProgress(row: { observation_json?: string | null }): Provisi
         if (v.blocker !== null && (!object(v.blocker) || typeof v.blocker.code !== 'string' || typeof v.blocker.message !== 'string'
             || typeof v.blocker.automatic !== 'boolean' || (v.blocker.href !== undefined && !nullableString(v.blocker.href))
             || (v.blocker.label !== undefined && !nullableString(v.blocker.label)))) return null;
-        if (v.checks.length > 10 || !v.checks.every(c => object(c) && ['container', 'game', 'map', 'occupancy', 'stopped'].includes(String(c.key))
+        if (v.checks.length > 10 || !v.checks.every(c => object(c) && ['container', 'game', 'map', 'plutonium-release', 'plutonium-version', 'occupancy', 'stopped'].includes(String(c.key))
             && typeof c.label === 'string' && ['pass', 'waiting', 'fail', 'unknown'].includes(String(c.result))
-            && (c.source === null || ['komodo', 'rcon', 'telemetry', 'combined'].includes(String(c.source)))
+            && (c.source === null || ['komodo', 'rcon', 'telemetry', 'combined', 'manifest'].includes(String(c.source)))
             && typeof c.checkedAt === 'string' && nullableString(c.receivedAt) && scalar(c.expected) && scalar(c.actual) && nullableString(c.message))) return null;
         if (v.checkpoints.length > 100 || !v.checkpoints.every(c => object(c) && typeof c.key === 'string' && typeof c.label === 'string'
             && typeof c.at === 'string' && (c.phase === null || phase(c.phase)))) return null;

src/lib/ui/components/ProvisioningProgress.svelte

@@ -9,7 +9,7 @@
     const progress = $derived(detail.progress);
     let expanded = $state<Record<string, boolean>>({});
     const evidenceLabels = { pass: 'Passed', waiting: 'Waiting', fail: 'Failed', unknown: 'Unknown' };
-    const sources = { komodo: 'Komodo', rcon: 'RCON', telemetry: 'Telemetry', combined: 'RCON and telemetry' };
+    const sources = { komodo: 'Komodo', rcon: 'RCON', telemetry: 'Telemetry', combined: 'RCON and telemetry', manifest: 'Plutonium manifest' };
 </script>
 
 <section class="execution" aria-labelledby="execution-title">

src/lib/ui/provisioning-presentation.ts

@@ -11,13 +11,14 @@ const phaseLabels: Record<ProvisioningPhase, string> = {
     prepare: 'Prepare server files', 'create-deployment': 'Create Komodo deployment', 'container-action': 'Start container',
     'remove-container': 'Stop and remove container', 'delete-deployment': 'Delete Komodo deployment', cleanup: 'Remove files and registration',
     'verify-container': 'Confirm container readiness', 'verify-game': 'Verify game status', 'verify-map': 'Verify selected engine map',
+    'verify-plutonium': 'Verify Plutonium revision',
     'verify-stopped': 'Confirm container is stopped'
 };
 type GroupDefinition = { key: string; label: string; phases: ProvisioningPhase[] };
 const request: GroupDefinition = { key: 'request', label: 'Request and eligibility', phases: ['requested', 'eligibility'] };
 const creation: GroupDefinition = { key: 'create', label: 'Create and start', phases: ['startup-queue', 'prepare', 'create-deployment', 'container-action'] };
 const removal: GroupDefinition = { key: 'remove', label: 'Remove the instance', phases: ['remove-container', 'delete-deployment', 'cleanup'] };
-const readiness: GroupDefinition = { key: 'verify', label: 'Verify readiness', phases: ['verify-container', 'verify-game', 'verify-map'] };
+const readiness: GroupDefinition = { key: 'verify', label: 'Verify readiness', phases: ['verify-container', 'verify-game', 'verify-map', 'verify-plutonium'] };
 function definitions(action: ProvisioningAction): GroupDefinition[] {
     if (action === 'create') return [{ ...request, phases: ['requested'] }, creation, readiness];
     if (action === 'rebuild') return [request, removal, { ...creation, label: 'Create a fresh instance' }, readiness];
@@ -36,7 +37,8 @@ export function provisioningGroups(detail: ProvisioningDetail): ProgressGroup[]
     return groups.map(group => {
         const steps = group.phases.map((key): ProgressStep => {
             const checkpoint = progress.checkpoints.find(item => item.key === key);
-            const checkKey = key === 'verify-container' ? 'container' : key === 'verify-game' ? 'game' : key === 'verify-map' ? 'map' : key === 'verify-stopped' ? 'stopped' : null;
+            const checkKey = key === 'verify-container' ? 'container' : key === 'verify-game' ? 'game' : key === 'verify-map' ? 'map'
+                : key === 'verify-plutonium' ? 'plutonium-version' : key === 'verify-stopped' ? 'stopped' : null;
             const check = checkKey ? progress.checks.find(item => item.key === checkKey) : null;
             const index = phases.indexOf(key);
             const isCurrent = key === progress.phase;
@@ -185,9 +187,9 @@ export function isProvisioningDetail(value: unknown, id: string): value is Provi
             && typeof progress.blocker.automatic === 'boolean' && optionalLink(progress.blocker.href)
             && (progress.blocker.label === undefined || nullableText(progress.blocker.label)))) return false;
         if (!Array.isArray(progress.checks) || !progress.checks.every(check => record(check)
-            && ['container', 'game', 'map', 'occupancy', 'stopped'].includes(check.key as string)
+            && ['container', 'game', 'map', 'plutonium-release', 'plutonium-version', 'occupancy', 'stopped'].includes(check.key as string)
             && ['pass', 'waiting', 'fail', 'unknown'].includes(check.result as string)
-            && [null, 'komodo', 'rcon', 'telemetry', 'combined'].includes(check.source as null | string)
+            && [null, 'komodo', 'rcon', 'telemetry', 'combined', 'manifest'].includes(check.source as null | string)
             && text(check.label) && timestamp(check.checkedAt) && nullableTime(check.receivedAt)
             && scalar(check.expected) && scalar(check.actual) && nullableText(check.message))) return false;
         if (!Array.isArray(progress.checkpoints) || !progress.checkpoints.every(checkpoint => record(checkpoint)

tests/unit/create-server.test.ts

@@ -16,6 +16,7 @@ vi.mock('../../src/lib/server/integrations/komodo.js', () => ({
 }));
 vi.mock('../../src/lib/server/integrations/rcon.js', () => ({
     rconGetPlayerCount: vi.fn(async () => 0),
+    rconPlutoniumVersion: vi.fn(async () => ({ revision: 5354, error: null })),
     rconStatusResult: vi.fn(async () => ({ ok: true, endpoint: { host: 'localhost', port: 27016 }, rawReply: '',
         status: { players: 0, map: 'zombie_cosmodrome', gametype: null, roster: [] } }))
 }));
@@ -29,6 +30,7 @@ beforeEach(async () => {
     vi.resetModules();
     vi.clearAllMocks();
     fixture = await newFixture([]);
+    vi.stubGlobal('fetch', vi.fn(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354, files: [] })));
     vi.stubEnv('KOMODO_ADDRESS', 'http://komodo.test');
     vi.stubEnv('KOMODO_API_KEY', 'test');
     vi.stubEnv('KOMODO_API_SECRET', 'test');
@@ -46,6 +48,7 @@ beforeEach(async () => {
 afterEach(async () => {
     await fixture.cleanup();
     vi.unstubAllEnvs();
+    vi.unstubAllGlobals();
 });
 
 describe('queued removal cancellation', () => {
@@ -486,7 +489,7 @@ describe('server creation map selection', () => {
         expect(getDeployment(id)?.deployment).toMatchObject({ state: 'awaiting-health' });
         vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: true, state: 'running' });
         await processCreations();
-        expect(getDeployment(id)?.deployment).toMatchObject({ state: 'done', finished_at: expect.any(String), stage: 'Verified Komodo healthy, RCON healthy, and map zombie_cosmodrome' });
+        expect(getDeployment(id)?.deployment).toMatchObject({ state: 'done', finished_at: expect.any(String), stage: 'Verified Komodo healthy, RCON healthy, map zombie_cosmodrome, and Plutonium r5354 (production r5354)' });
     });
 
     it('returns a deployment URL identifier immediately and serves its lifecycle detail', async () => {

tests/unit/creation-queue.test.ts

@@ -14,6 +14,7 @@ vi.mock('../../src/lib/server/integrations/komodo.js', () => ({
 }));
 vi.mock('../../src/lib/server/integrations/rcon.js', () => ({
     rconGetPlayerCount: vi.fn(),
+    rconPlutoniumVersion: vi.fn(),
     rconStatusResult: vi.fn()
 }));
 vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}) }));
@@ -28,6 +29,7 @@ function status(server: string, map = server.includes('_moon_') ? 'zombie_moon'
 beforeEach(async () => {
     vi.resetModules(); vi.resetAllMocks();
     fixture = await newFixture([]);
+    vi.stubGlobal('fetch', vi.fn(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354, files: [] })));
     vi.stubEnv('KOMODO_ADDRESS', 'http://komodo.test');
     vi.stubEnv('KOMODO_API_KEY', 'test-key');
     vi.stubEnv('KOMODO_API_SECRET', 'test-secret');
@@ -46,11 +48,12 @@ beforeEach(async () => {
     vi.mocked(komodo.komodoUpdateTagsOnResource).mockResolvedValue({ ok: true });
     const rcon = await import('../../src/lib/server/integrations/rcon.js');
     vi.mocked(rcon.rconGetPlayerCount).mockResolvedValue(0);
+    vi.mocked(rcon.rconPlutoniumVersion).mockResolvedValue({ revision: 5354, error: null });
     vi.mocked(rcon.rconStatusResult).mockImplementation(async server => status(server));
     const { kick } = await import('../../src/lib/server/pipeline/worker.js');
     vi.mocked(kick).mockResolvedValue(undefined);
 });
-afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); vi.restoreAllMocks(); });
+afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); vi.unstubAllGlobals(); vi.restoreAllMocks(); });
 
 async function queueServers(count: number) {
     const { queueCreateServer } = await import('../../src/lib/server/fleet/create.js');

tests/unit/deployment-collections.test.ts

@@ -15,6 +15,7 @@ vi.mock('../../src/lib/server/integrations/komodo.js', () => ({
 }));
 vi.mock('../../src/lib/server/integrations/rcon.js', () => ({
     rconGetPlayerCount: vi.fn(async () => 0),
+    rconPlutoniumVersion: vi.fn(async () => ({ revision: 5354, error: null })),
     rconStatusResult: vi.fn(async (server: string) => ({ ok: true, endpoint: { host: 'localhost', port: 28960 }, rawReply: '',
         status: { players: 0, map: server.startsWith('t6_') ? 'zm_transit' : server.includes('_moon_') ? 'zombie_moon' : 'zombie_theater', gametype: null, roster: [] } }))
 }));
@@ -29,6 +30,7 @@ const entries = [
 beforeEach(async () => {
     vi.resetModules(); vi.clearAllMocks();
     fixture = await newFixture([]);
+    vi.stubGlobal('fetch', vi.fn(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354, files: [] })));
     vi.stubEnv('KOMODO_ADDRESS', 'http://komodo.test');
     vi.stubEnv('KOMODO_API_KEY', 'private-api-key');
     vi.stubEnv('KOMODO_API_SECRET', 'private-api-secret');
@@ -38,7 +40,7 @@ beforeEach(async () => {
     vi.stubEnv('SERVER_PORT_MAX', '28980');
     vi.stubEnv('DEPLOYMENT_CREATE_CONCURRENCY', '1');
 });
-afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); });
+afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); vi.unstubAllGlobals(); });
 
 async function previewLineup() {
     const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');

tests/unit/launch-recipes.test.ts

@@ -14,6 +14,7 @@ vi.mock('../../src/lib/server/integrations/komodo.js', () => ({
 }));
 vi.mock('../../src/lib/server/integrations/rcon.js', () => ({
     rconGetPlayerCount: vi.fn(async () => 0),
+    rconPlutoniumVersion: vi.fn(async () => ({ revision: 5354, error: null })),
     rconStatusResult: vi.fn(async () => ({ ok: true, endpoint: { host: 'localhost', port: 28960 }, rawReply: '',
         status: { players: 0, map: 'zombie_theater', gametype: null, roster: [] } }))
 }));
@@ -26,13 +27,14 @@ const input = { ...selection, gameMode: 'zombie' as const, plutoKey: 'private-pl
 beforeEach(async () => {
     vi.resetModules(); vi.clearAllMocks();
     fixture = await newFixture([]);
+    vi.stubGlobal('fetch', vi.fn(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354, files: [] })));
     vi.stubEnv('KOMODO_ADDRESS', 'http://komodo.test');
     vi.stubEnv('KOMODO_API_KEY', 'private-api-key');
     vi.stubEnv('KOMODO_API_SECRET', 'private-api-secret');
     vi.stubEnv('KOMODO_TARGET_SERVER', 'test-host');
     vi.stubEnv('RCON_PASSWORD', 'private-rcon-password');
 });
-afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); });
+afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); vi.unstubAllGlobals(); });
 
 async function seedArtifacts() {
     const { createScript } = await import('../../src/lib/server/fleet/scripts.js');

tests/unit/plutonium.test.ts

@@ -31,6 +31,35 @@ afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); vi.unstubAl
 const api = () => import('../../src/lib/server/fleet/plutonium.js');
 
 describe('Plutonium versions', () => {
+    it('coalesces published-release checks without probing the fleet', async () => {
+        const { checkRelease, release } = await import('../../src/lib/server/integrations/plutonium.js');
+        const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
+        const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
+        const first = checkRelease();
+        const second = checkRelease();
+        expect(second).toBe(first);
+        await Promise.all([first, second]);
+        expect(fetch).toHaveBeenCalledTimes(1);
+        expect(release()).toMatchObject({ revision: 5354, error: null });
+        expect(komodoInspectHealthy).not.toHaveBeenCalled();
+        expect(rconPlutoniumVersion).not.toHaveBeenCalled();
+        await checkRelease();
+        expect(fetch).toHaveBeenCalledTimes(2);
+    });
+    it('refreshes hourly fleet versions even when provisioning recently refreshed the release manifest', async () => {
+        const m = await api();
+        const { checkRelease, HOUR } = await import('../../src/lib/server/integrations/plutonium.js');
+        const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
+        await m.checkPlutonium();
+        vi.setSystemTime(Date.now() + HOUR);
+        await checkRelease();
+        vi.mocked(rconPlutoniumVersion).mockClear();
+        await m.maintenanceTick();
+        expect(rconPlutoniumVersion).toHaveBeenCalledTimes(servers.length);
+        for (const server of servers) expect(rconPlutoniumVersion).toHaveBeenCalledWith(server);
+        await m.maintenanceTick();
+        expect(rconPlutoniumVersion).toHaveBeenCalledTimes(servers.length);
+    });
     it('uses the official production manifest without requesting the challenged changelog page', async () => {
         const m = await api(); await m.checkPlutonium();
         expect(fetch).toHaveBeenCalledTimes(1);
@@ -68,6 +97,78 @@ describe('Plutonium versions', () => {
         m.queueMaintenance([servers[0]]);
         expect(m.plutoniumStatus().jobs[0].target_revision).toBeNull();
     });
+    it('rejects a version observed at the startup submission timestamp, then accepts a later observation', async () => {
+        const { db } = await import('../../src/lib/server/db/index.js');
+        const { probeVersion, runningVersions, checkRelease } = await import('../../src/lib/server/integrations/plutonium.js');
+        const submittedAt = new Date().toISOString();
+        db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, submitted_at, triggered_by)
+            VALUES ('startup', ?, 'create', 'awaiting-health', 'Waiting for health', ?, ?, 'test')`).run(servers[0], submittedAt, submittedAt);
+        await checkRelease();
+        expect(await probeVersion(servers[0])).toBeNull();
+        expect(runningVersions()[0]).toMatchObject({ revision: null, checked_at: submittedAt, update_status: 'unknown', error: expect.stringContaining('after startup') });
+        vi.setSystemTime(Date.now() + 1);
+        expect(await probeVersion(servers[0])).toBe(5354);
+        expect(runningVersions()[0]).toMatchObject({ revision: 5354, update_status: 'current', error: null });
+    });
+    it('discards a running revision when a new lifecycle submission arrives during its probe', async () => {
+        const { db } = await import('../../src/lib/server/db/index.js');
+        const { probeVersion, runningVersions } = await import('../../src/lib/server/integrations/plutonium.js');
+        const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
+        vi.mocked(rconPlutoniumVersion).mockImplementationOnce(async () => {
+            const now = new Date().toISOString();
+            db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, submitted_at, triggered_by)
+                VALUES ('restart', ?, 'restart', 'awaiting-health', 'Waiting for health', ?, ?, 'test')`).run(servers[0], now, now);
+            return { revision: 5354, error: null };
+        });
+        expect(await probeVersion(servers[0])).toBeNull();
+        expect(runningVersions()[0]).toMatchObject({ revision: null, update_status: 'unknown' });
+        expect(db().prepare('SELECT * FROM plutonium_versions WHERE server = ?').get(servers[0])).toBeUndefined();
+    });
+    it.each(['startup', 'replacement'] as const)('preserves a newer successful revision when an old probe finishes after %s', async change => {
+        const { db } = await import('../../src/lib/server/db/index.js');
+        const { probeVersion, runningVersions, checkRelease } = await import('../../src/lib/server/integrations/plutonium.js');
+        const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
+        await checkRelease();
+        db().prepare('UPDATE servers SET created_at = ? WHERE name = ?').run('2026-09-21T05:58:00.000Z', servers[0]);
+        let finishOld!: (value: { revision: number; error: null }) => void;
+        let started!: () => void;
+        const probing = new Promise<void>(resolve => { started = resolve; });
+        vi.mocked(rconPlutoniumVersion).mockImplementationOnce(() => {
+            started();
+            return new Promise(resolve => { finishOld = resolve; });
+        });
+        const oldProbe = probeVersion(servers[0]);
+        await probing;
+        const now = new Date().toISOString();
+        if (change === 'startup') {
+            db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, submitted_at, triggered_by)
+                VALUES ('new-startup', ?, 'start', 'awaiting-health', 'Waiting for health', ?, ?, 'test')`).run(servers[0], now, now);
+        } else {
+            // Both registrations have no lifecycle submission, so only their
+            // registry identity distinguishes the old and replacement instances.
+            db().prepare('DELETE FROM servers WHERE name = ?').run(servers[0]);
+            db().prepare('INSERT INTO servers (name, komodo_name, created_at) VALUES (?, ?, ?)').run(servers[0], servers[0], now);
+        }
+        vi.setSystemTime(Date.now() + 1);
+        expect(await probeVersion(servers[0])).toBe(5354);
+        const saved = db().prepare('SELECT * FROM plutonium_versions WHERE server = ?').get(servers[0]);
+        vi.setSystemTime(Date.now() + 1);
+        finishOld({ revision: 5300, error: null });
+        expect(await oldProbe).toBeNull();
+        expect(db().prepare('SELECT * FROM plutonium_versions WHERE server = ?').get(servers[0])).toEqual(saved);
+        expect(runningVersions()[0]).toMatchObject({ revision: 5354, error: null, update_status: 'current' });
+    });
+    it('discards a running revision when the server is stopped during its probe', async () => {
+        const { db } = await import('../../src/lib/server/db/index.js');
+        const { probeVersion, runningVersions } = await import('../../src/lib/server/integrations/plutonium.js');
+        const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
+        vi.mocked(rconPlutoniumVersion).mockImplementationOnce(async () => {
+            db().prepare("UPDATE servers SET desired_state = 'stopped' WHERE name = ?").run(servers[0]);
+            return { revision: 5354, error: null };
+        });
+        expect(await probeVersion(servers[0])).toBeNull();
+        expect(runningVersions()[0]).toMatchObject({ revision: null, error: 'Server is intentionally stopped.', update_status: 'stopped' });
+    });
 });
 
 describe('maintenance timezones', () => {

tests/unit/provisioning-evidence.test.ts

@@ -8,7 +8,7 @@ vi.mock('../../src/lib/server/integrations/komodo.js', () => ({
     komodoCreateDeployment: vi.fn(), komodoDeploy: vi.fn(), komodoDestroyDeployment: vi.fn(), komodoDeleteDeployment: vi.fn(),
     komodoEnsureTag: vi.fn(), komodoUpdateTagsOnResource: vi.fn()
 }));
-vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconGetPlayerCount: vi.fn(), rconStatusResult: vi.fn() }));
+vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconGetPlayerCount: vi.fn(), rconStatusResult: vi.fn(), rconPlutoniumVersion: vi.fn() }));
 vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}) }));
 
 let fixture: Fixture;
@@ -20,6 +20,7 @@ beforeEach(async () => {
     vi.resetModules(); vi.resetAllMocks();
     vi.useFakeTimers({ toFake: ['Date'] });
     vi.setSystemTime(new Date('2026-09-28T12:00:00Z'));
+    vi.stubGlobal('fetch', vi.fn(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354 })));
     fixture = await newFixture([server]);
     for (const [key, value] of Object.entries({ KOMODO_ADDRESS: 'http://komodo.test', KOMODO_API_KEY: 'test', KOMODO_API_SECRET: 'test',
         KOMODO_TARGET_SERVER: 'host', RCON_PASSWORD: 'test', DEPLOYMENT_CREATE_CONCURRENCY: '1', DEPLOYMENT_RESTART_CONCURRENCY: '1',
@@ -34,8 +35,9 @@ beforeEach(async () => {
     const rcon = await import('../../src/lib/server/integrations/rcon.js');
     vi.mocked(rcon.rconGetPlayerCount).mockResolvedValue(0);
     vi.mocked(rcon.rconStatusResult).mockResolvedValue(status());
+    vi.mocked(rcon.rconPlutoniumVersion).mockResolvedValue({ revision: 5354, error: null });
 });
-afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); vi.unstubAllEnvs(); });
+afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); vi.unstubAllEnvs(); vi.unstubAllGlobals(); });
 
 async function register() {
     const { db } = await import('../../src/lib/server/db/index.js');
@@ -57,6 +59,125 @@ async function telemetry(players: number, map = 'zombie_theater') {
     return receivedAt;
 }
 
+async function launch(game: 't5' | 't6' = 't5') {
+    const { queueCreateServer, processCreations } = await import('../../src/lib/server/provisioning/instances.js');
+    const { rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js');
+    vi.mocked(rconStatusResult).mockResolvedValue(status(game === 't5' ? 'zombie_theater' : 'zm_transit'));
+    const job = queueCreateServer({ game, gameMode: 'zombie', mapSlug: game === 't5' ? 'theater' : 'town', plutoKey: 'saved-key' });
+    if (!job.ok) throw new Error(job.message);
+    await processCreations();
+    vi.setSystemTime(Date.now() + 15_000);
+    return { ...job, processCreations };
+}
+
+describe('Plutonium provisioning verification', () => {
+    it.each(['t5', 't6'] as const)('verifies production and persists the running %s revision before completing', async game => {
+        const job = await launch(game);
+        const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
+        const { runningVersions } = await import('../../src/lib/server/integrations/plutonium.js');
+        const { PLUTONIUM_MANIFEST_URL } = await import('../../src/lib/plutonium.js');
+        expect(rconPlutoniumVersion).not.toHaveBeenCalled();
+        await job.processCreations();
+        const completed = await detail(job.deploymentId);
+        expect(completed.deployment.state).toBe('done');
+        expect(completed.progress).toMatchObject({ phase: 'verify-plutonium', blocker: null });
+        expect(completed.progress?.checks).toEqual(expect.arrayContaining([
+            expect.objectContaining({ key: 'plutonium-release', source: 'manifest', result: 'pass', actual: 5354 }),
+            expect.objectContaining({ key: 'plutonium-version', source: 'rcon', result: 'pass', expected: 5354, actual: 5354 })
+        ]));
+        expect(runningVersions().find(version => version.server === job.name)).toMatchObject({ revision: 5354, update_status: 'current' });
+        expect(fetch).toHaveBeenCalledWith(PLUTONIUM_MANIFEST_URL, expect.objectContaining({ redirect: 'error' }));
+        expect(rconPlutoniumVersion).toHaveBeenCalledTimes(1);
+        expect(rconPlutoniumVersion).toHaveBeenCalledWith(job.name);
+        const { isProvisioningDetail, provisioningGroups } = await import('../../src/lib/ui/provisioning-presentation.js');
+        expect(isProvisioningDetail(completed, job.deploymentId)).toBe(true);
+        expect(provisioningGroups(completed).find(group => group.key === 'verify')?.state).toBe('done');
+    });
+
+    it('retries unknown runtime evidence while retaining startup capacity and never replays creation', async () => {
+        const job = await launch();
+        const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
+        const { komodoCreateDeployment, komodoDeploy } = await import('../../src/lib/server/integrations/komodo.js');
+        const { queueCreateServer } = await import('../../src/lib/server/provisioning/instances.js');
+        const next = queueCreateServer({ gameMode: 'zombie', mapSlug: 'theater', plutoKey: 'second-key' });
+        if (!next.ok) throw new Error(next.message);
+        // A pre-start observation is not proof of the newly started process.
+        const { db } = await import('../../src/lib/server/db/index.js');
+        db().prepare('INSERT INTO plutonium_versions (server, revision, checked_at) VALUES (?, 5354, ?)')
+            .run(job.name, '2026-09-28T11:59:00.000Z');
+        vi.mocked(rconPlutoniumVersion).mockResolvedValueOnce({ revision: null, error: 'Version reply timed out.' });
+        await job.processCreations();
+        expect((await detail(job.deploymentId)).deployment.state).toBe('awaiting-health');
+        expect((await detail(job.deploymentId)).progress?.blocker).toMatchObject({ code: 'plutonium-version', automatic: true });
+        expect((await detail(next.deploymentId)).deployment.state).toBe('queued');
+        expect(db().prepare('SELECT revision FROM plutonium_versions WHERE server = ?').get(job.name)).toEqual({ revision: null });
+        await job.processCreations();
+        expect((await detail(job.deploymentId)).deployment.state).toBe('done');
+        expect((await detail(next.deploymentId)).deployment.state).toBe('awaiting-health');
+        expect(komodoCreateDeployment).toHaveBeenCalledTimes(2);
+        expect(komodoDeploy).toHaveBeenCalledTimes(2);
+    });
+
+    it('requires a successful manifest check even with a previously cached revision, then resumes without recreating', async () => {
+        const job = await launch();
+        const { writeSetting } = await import('../../src/lib/server/maintenance/settings.js');
+        writeSetting('plutonium_release', { revision: 5300, checkedAt: new Date().toISOString(), error: null });
+        vi.mocked(fetch).mockRejectedValueOnce(new Error('Manifest offline'));
+        await job.processCreations();
+        expect((await detail(job.deploymentId)).deployment.state).toBe('awaiting-health');
+        expect((await detail(job.deploymentId)).progress?.checks).toContainEqual(expect.objectContaining({
+            key: 'plutonium-release', result: 'unknown', actual: null, message: 'Manifest offline'
+        }));
+        await job.processCreations();
+        expect((await detail(job.deploymentId)).deployment.state).toBe('done');
+        const { komodoDeploy } = await import('../../src/lib/server/integrations/komodo.js');
+        expect(komodoDeploy).toHaveBeenCalledTimes(1);
+    });
+
+    it('reports a confirmed outdated runtime without automatically restarting or recreating it', async () => {
+        const job = await launch();
+        const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
+        vi.mocked(rconPlutoniumVersion).mockResolvedValue({ revision: 5300, error: null });
+        await job.processCreations();
+        const failed = await detail(job.deploymentId);
+        expect(failed.deployment).toMatchObject({ state: 'failed', error: expect.stringContaining('running r5300, production is r5354') });
+        expect(failed.progress?.checks).toContainEqual(expect.objectContaining({ key: 'plutonium-version', result: 'fail', expected: 5354, actual: 5300 }));
+        const { runningVersions } = await import('../../src/lib/server/integrations/plutonium.js');
+        expect(runningVersions().find(version => version.server === job.name)).toMatchObject({ revision: 5300, update_status: 'outdated' });
+        await job.processCreations();
+        const { komodoCreateDeployment, komodoDeploy, komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
+        expect(komodoCreateDeployment).toHaveBeenCalledTimes(1);
+        expect(komodoDeploy).toHaveBeenCalledTimes(1);
+        expect(komodoRestart).not.toHaveBeenCalled();
+    });
+
+    it('accepts a runtime ahead of the published release without requesting a downgrade', async () => {
+        const job = await launch();
+        const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
+        vi.mocked(rconPlutoniumVersion).mockResolvedValue({ revision: 5355, error: null });
+        await job.processCreations();
+        expect((await detail(job.deploymentId)).deployment.state).toBe('done');
+        const { runningVersions } = await import('../../src/lib/server/integrations/plutonium.js');
+        expect(runningVersions().find(version => version.server === job.name)?.update_status).toBe('ahead');
+    });
+
+    it('discards a late version response when the instance was removed during the probe', async () => {
+        const job = await launch();
+        const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
+        const { db } = await import('../../src/lib/server/db/index.js');
+        vi.mocked(rconPlutoniumVersion).mockImplementationOnce(async () => {
+            db().prepare('DELETE FROM servers WHERE name = ?').run(job.name);
+            return { revision: 5354, error: null };
+        });
+        await job.processCreations();
+        const removed = await detail(job.deploymentId);
+        expect(removed.deployment.server_deleted_at).toBeTruthy();
+        expect(removed.deployment.state).not.toBe('done');
+        expect(removed.progress?.checkpoints.some(check => check.key === 'verify-plutonium')).toBe(false);
+        expect(db().prepare('SELECT * FROM plutonium_versions WHERE server = ?').get(job.name)).toBeUndefined();
+    });
+});
+
 describe('provisioning decision evidence', () => {
     it('distinguishes queued recipes, attempted files, transient map checks, and telemetry completion without replaying startup', async () => {
         const { queueCreateServer, processCreations } = await import('../../src/lib/server/provisioning/instances.js');

tests/unit/provisioning-presentation.test.ts

@@ -81,6 +81,36 @@ describe('provisioning outcome and evidence presentation', () => {
         expect(provisioningGroups(detail).flatMap(group => group.steps).find(step => step.key === 'verify-game')).toMatchObject({ state: 'unknown', at: '2026-09-28T12:01:00Z' });
     });
 
+    it.each(['create', 'rebuild'] as const)('shows %s Plutonium verification from runtime evidence, separately from the release manifest', action => {
+        const detail = fixture();
+        detail.deployment.action = action;
+        detail.progress = progress({ phase: 'verify-plutonium',
+            checks: [
+                { key: 'plutonium-release', label: 'Production Plutonium release', result: 'pass', source: 'manifest', checkedAt: at, receivedAt: at, expected: null, actual: 5200, message: null },
+                { key: 'plutonium-version', label: 'Running Plutonium revision', result: 'unknown', source: 'rcon', checkedAt: at, receivedAt: null, expected: 5200, actual: null, message: 'Waiting for a runtime revision' }
+            ] });
+        expect(isProvisioningDetail(detail, detail.deployment.id)).toBe(true);
+        for (const [result, state] of [['unknown', 'unknown'], ['waiting', 'waiting'], ['fail', 'failed'], ['pass', 'done']] as const) {
+            detail.progress.checks[1].result = result;
+            const steps = provisioningGroups(detail).find(group => group.key === 'verify')!.steps;
+            expect(steps.at(-1)).toMatchObject({ key: 'verify-plutonium', label: 'Verify Plutonium revision', state, at });
+        }
+    });
+
+    it.each(['start', 'restart', 'stop', 'destroy'] as const)('does not add Plutonium verification to %s', action => {
+        const detail = fixture();
+        detail.deployment.action = action;
+        expect(provisioningGroups(detail).flatMap(group => group.steps).some(step => step.key === 'verify-plutonium')).toBe(false);
+    });
+
+    it('does not claim historical creations verified Plutonium without evidence', () => {
+        const detail = fixture();
+        detail.deployment.state = 'done';
+        detail.progress = progress({ checkpoints: [{ key: 'verify-map', label: 'Selected map verified', at, phase: 'verify-map' }] });
+        expect(provisioningGroups(detail).flatMap(group => group.steps).find(step => step.key === 'verify-plutonium'))
+            .toMatchObject({ state: 'unknown', at: null });
+    });
+
     it('preserves the failed request when later recovery checks pass', () => {
         const detail = fixture();
         detail.deployment.action = 'restart'; detail.deployment.state = 'failed';

tests/unit/provisioning-progress.test.ts

@@ -54,6 +54,22 @@ describe('provisioning evidence persistence', () => {
         expect(handle.prepare("SELECT message FROM server_deployment_events WHERE kind = 'command'").get()).toEqual({ message: 'Reply was lost' });
     });
 
+    it('preserves Plutonium release and runtime evidence when later progress is recorded', async () => {
+        const { recordProgress, read } = await setup();
+        recordProgress('operation', { phase: 'verify-plutonium',
+            checks: [
+                { key: 'plutonium-release', label: 'Production Plutonium release', result: 'pass', source: 'manifest', actual: 5200 },
+                { key: 'plutonium-version', label: 'Running Plutonium revision', result: 'pass', source: 'rcon', expected: 5200, actual: 5200 }
+            ], checkpoint: { key: 'verify-plutonium', label: 'Plutonium revision verified' } });
+        vi.setSystemTime(Date.now() + 15_000);
+        recordProgress('operation', { blocker: null });
+        expect(read()).toMatchObject({ phase: 'verify-plutonium', phaseStartedAt: '2026-09-28T12:00:00.000Z',
+            checks: [
+                { key: 'plutonium-release', result: 'pass', source: 'manifest', actual: 5200 },
+                { key: 'plutonium-version', result: 'pass', source: 'rcon', expected: 5200, actual: 5200 }
+            ], checkpoints: [{ key: 'verify-plutonium', phase: 'verify-plutonium' }] });
+    });
+
     it('keeps operation-owned recovery evidence after slot/server removal and does not rewrite the failed outcome', async () => {
         const { handle, recordProgress, read } = await setup();
         handle.prepare("UPDATE server_deployments SET state = 'failed', finished_at = '2026-09-28T12:00:00Z' WHERE id = 'operation'").run();

tests/unit/provisioning.test.ts

@@ -10,7 +10,7 @@ vi.mock('../../src/lib/server/integrations/komodo.js', () => ({
     komodoEnsureTag: vi.fn(async (name: string) => ({ ok: true, id: name })),
     komodoUpdateTagsOnResource: vi.fn(async () => ({ ok: true }))
 }));
-vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconGetPlayerCount: vi.fn(), rconStatusResult: vi.fn() }));
+vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconGetPlayerCount: vi.fn(), rconStatusResult: vi.fn(), rconPlutoniumVersion: vi.fn() }));
 vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}) }));
 
 let fixture: Fixture;
@@ -23,6 +23,7 @@ beforeEach(async () => {
     vi.useFakeTimers({ toFake: ['Date'] });
     vi.setSystemTime(new Date('2026-09-27T12:00:00Z'));
     fixture = await newFixture([server]);
+    vi.stubGlobal('fetch', vi.fn(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354, files: [] })));
     for (const [key, value] of Object.entries({ KOMODO_ADDRESS: 'http://komodo.test', KOMODO_API_KEY: 'test', KOMODO_API_SECRET: 'test',
         KOMODO_TARGET_SERVER: 'host', RCON_PASSWORD: 'test', DEPLOYMENT_CREATE_CONCURRENCY: '1', DEPLOYMENT_RESTART_CONCURRENCY: '1',
         SERVER_PORT_MIN: '28960', SERVER_PORT_MAX: '28970' })) vi.stubEnv(key, value);
@@ -52,8 +53,9 @@ beforeEach(async () => {
     const rcon = await import('../../src/lib/server/integrations/rcon.js');
     vi.mocked(rcon.rconGetPlayerCount).mockResolvedValue(0);
     vi.mocked(rcon.rconStatusResult).mockResolvedValue(healthyRcon);
+    vi.mocked(rcon.rconPlutoniumVersion).mockResolvedValue({ revision: 5354, error: null });
 });
-afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); vi.unstubAllEnvs(); });
+afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); vi.unstubAllEnvs(); vi.unstubAllGlobals(); });
 
 async function operation(id: string) {
     const { db } = await import('../../src/lib/server/db/index.js');
@@ -174,8 +176,11 @@ describe('provisioning operations', () => {
         expect(await operation(id)).toMatchObject({ state: 'awaiting-health' });
         expect(await readFile(join(fixture.t5, server, 'runtime/plutonium/storage/t5/raw/scripts/sp/fresh.gsc'), 'utf8')).toBe('fresh content');
         vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: true, state: 'running' });
+        vi.setSystemTime(Date.now() + 1);
         await processCreations();
         expect(await operation(id)).toMatchObject({ state: 'done' });
+        const { runningVersions } = await import('../../src/lib/server/integrations/plutonium.js');
+        expect(runningVersions().find(version => version.server === server)).toMatchObject({ revision: 5354, update_status: 'current' });
     });
 
     it('rejects a rebuild without a saved key before touching the existing instance', async () => {

tests/unit/watchdog.test.ts

@@ -9,7 +9,7 @@ vi.mock('../../src/lib/server/integrations/komodo.js', () => ({
     komodoCreateDeployment: vi.fn(), komodoDeploy: vi.fn(), komodoDestroyDeployment: vi.fn(), komodoDeleteDeployment: vi.fn(),
     komodoEnsureTag: vi.fn(), komodoUpdateTagsOnResource: vi.fn()
 }));
-vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconGetPlayerCount: vi.fn(), rconStatusResult: vi.fn() }));
+vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconGetPlayerCount: vi.fn(), rconStatusResult: vi.fn(), rconPlutoniumVersion: vi.fn() }));
 vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}) }));
 
 const server = 't5_zombie_theater_1';
@@ -28,6 +28,7 @@ beforeEach(async () => {
     vi.useFakeTimers({ toFake: ['Date'] });
     vi.setSystemTime(new Date('2026-09-27T12:00:00Z'));
     fixture = await newFixture([server]);
+    vi.stubGlobal('fetch', vi.fn(async () => Response.json({ product: 'plutonium-core-prod', revision: 5354, files: [] })));
     for (const [key, value] of Object.entries({ KOMODO_ADDRESS: 'http://komodo.test', KOMODO_API_KEY: 'test', KOMODO_API_SECRET: 'test',
         KOMODO_TARGET_SERVER: 'host', RCON_PASSWORD: 'test', DEPLOYMENT_CREATE_CONCURRENCY: '1', DEPLOYMENT_RESTART_CONCURRENCY: '1',
         SERVER_PORT_MIN: '28960', SERVER_PORT_MAX: '28970' })) vi.stubEnv(key, value);
@@ -47,9 +48,10 @@ beforeEach(async () => {
     rcon = await import('../../src/lib/server/integrations/rcon.js');
     vi.mocked(rcon.rconGetPlayerCount).mockResolvedValue(0);
     vi.mocked(rcon.rconStatusResult).mockResolvedValue(healthyRcon);
+    vi.mocked(rcon.rconPlutoniumVersion).mockResolvedValue({ revision: 5354, error: null });
     watchdog = await import('../../src/lib/server/watchdog/index.js');
 });
-afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); vi.unstubAllEnvs(); });
+afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); vi.unstubAllEnvs(); vi.unstubAllGlobals(); });
 
 const incidents = () => watchdog.watchdogOverview().incidents;
 const onlyIncident = () => { expect(incidents()).toHaveLength(1); return incidents()[0]; };
@@ -375,6 +377,7 @@ describe('Watchdog drift campaigns', () => {
             .toEqual({ provisioning_id: action.operation_id, restart_state: 'pending' });
         await processCreations();
         vi.mocked(komodo.komodoInspectHealthy).mockResolvedValue({ ok: true, state: 'running' });
+        advance(1);
         await processCreations(); await sweep(); await watchdog.watchdogTick(true);
         expect(watchdog.watchdogDetail(row.id)?.incident.state).toBe('resolved');
         expect(database.db().prepare('SELECT restart_state FROM deployment_servers WHERE deployment_id = ?').get(action.campaign_id)).toEqual({ restart_state: 'done' });