Add safe manual instance rebalancing

docs/README.md

@@ -17,6 +17,7 @@ and launch your first server.
 - [Provisioning](guides/provisioning.md) — lifecycle actions, safety checks, queues, and diagnostics.
 - [Collections](guides/collections.md) — reusable map lineups and promoted launch files.
 - [Lobby autoscaling](guides/autoscaling.md) — policies, ownership, pause controls, and the pilot procedure.
+- [Rebalancing](guides/native-hosts.md#rebalance-instances) — preview and safely spread empty lobbies across active hosts.
 - [Maintenance](guides/maintenance.md) — health recovery, drift repair, updates, and scheduled restarts.
 
 ### Releases and analytics

docs/guides/autoscaling.md

@@ -29,6 +29,11 @@ A live policy owns the lobby counts for its selected maps. Overlapping live game
 
 Unknown or stale occupancy, stopped or unhealthy members, active betas, campaigns, maintenance, host drains, and unresolved recovery hold the policy's decisions. Autoscaling does not repair these conditions itself. The page and linked Provisioning operation explain the current blocker, including capacity and game key waits.
 
+[Capacity rebalancing](native-hosts.md#rebalance-instances) preserves lobby identities
+and counts. Each move uses a Provisioning rebuild, which holds scaling for the
+affected map until the move settles. The temporary outage does not request a
+replacement lobby.
+
 Current positive occupancy from either telemetry or RCON vetoes an empty claim. Empty intervals reset when players arrive, evidence becomes unknown, the instance or lifecycle changes, or observation continuity breaks. Controller downtime does not count toward the empty duration. Failed or uncertain automatic operations pause further actions until an operator inspects their Provisioning recovery and reviews the policy again.
 
 ## Pause and cancellation

docs/guides/native-hosts.md

@@ -442,6 +442,70 @@ restart and create complete only after post-operation health evidence; creation
 and rebuild also verify the selected map and running Plutonium revision. The
 base-game library and runtime revision are independent records.
 
+## Rebalance instances
+
+Open **Fleet → Capacity → Rebalance** to spread existing instances across active
+hosts. Hosts also provides a shortcut. Operators and administrators can run a
+rebalance; viewers can inspect its history.
+
+Choose at least two participating hosts, a game selection, and a maximum of
+1–20 moves (initially five), then choose **Preview rebalance**. The preview shows
+current and projected instance counts, CPU and memory utilization, proposed
+source/destination pairs, and exclusions. A larger host can carry more instances:
+the planner balances resource utilization relative to each host's capacity.
+The selected game limits which instances can move; the estimates include all
+workloads on the participating hosts.
+
+The planner uses current measurements and conservative five-minute host history,
+current-generation instance resource samples, destination running/startup
+allowances, and existing reservations. Each move must reduce the source/destination
+pair's peak CPU-or-memory utilization by at least five percentage points. A
+successfully moved instance is excluded for 30 minutes to limit repeated moves.
+Unplaced queued launches do not reserve physical resources.
+
+Only healthy instances with fresh, confirmed empty occupancy and recoverable
+deployed files qualify. Occupied, stopped, unknown, stale, and incompatible
+instances are excluded, as are beta, campaign, maintenance, and unresolved
+recovery work. Positive current occupancy from either RCON or telemetry vetoes
+an empty claim. A partially balanced result is normal when eligible instances
+cannot improve the distribution enough.
+
+Choose **Start rebalance** within five minutes of the preview. Submission
+rechecks the proposed pairs; a changed plan requires a new preview. One durable
+run executes at a time, and it never adds moves beyond the approved list. Before
+each move it rechecks occupancy, identity, capacity, and competing work. A move
+that no longer qualifies is skipped. After each healthy completion the run waits
+for fresh reports from both affected hosts before considering its next move.
+
+Provisioning reserves the destination before removing the source, then recreates
+and health-verifies the lobby. The source host stays active. This is an empty-lobby
+rebuild with a temporary outage; its address or port may change. The logical
+identity, map, saved key, runtime credentials, runtime image, game-file library,
+and recorded deployed configuration/scripts/plugins are retained. Untracked
+local mutable files are not copied. If deployed bytes or credentials are no longer
+available, the instance is excluded instead of silently upgrading to promoted files.
+
+**Stop remaining moves** withdraws work that has not committed source removal.
+Once source removal has been queued, that move continues through destination
+creation and health verification. Stopping does not undo completed moves.
+Unknown agent results retain their original operation and reservations until
+reconciliation; they never authorize a duplicate instance. Failures stop the
+remaining moves and link to Provisioning recovery. Recovering a move there does
+not resume the rest of its run. Runs and operation links survive controller
+restarts; completed/stopped runs are retained for 31 days.
+
+Autoscaling retains ownership of lobby counts. A move is represented as a rebuild,
+so the affected map's scaling decisions wait for it to settle instead of treating
+the temporary outage as missing capacity.
+
+`GET /api/rebalancing` returns host choices and the 20 most recent runs, with active
+runs first. `POST /api/rebalancing` accepts `preview` with `settings` (`hostIds`,
+`game: all|t5|t6`, `maxMoves`), `start` with `previewId`, or `stop` with the run `id`.
+Writes use the existing operator authorization and same-origin checks. Repeating
+Start with the same preview returns its original run. The controller planner and
+run state live in `src/lib/server/rebalancing/`; all host mutations remain in
+Provisioning. Rebalancing is manual; there is no periodic automatic policy.
+
 ## Drain and enable hosts
 
 **Drain host** asks for confirmation, blocks new deployments, and migrates

docs/guides/provisioning.md

@@ -12,6 +12,13 @@ are shown. Confirmations name the server and its current players and round;
 disruptive actions still wait for a fresh empty-lobby check. **Manage server**
 on the operations page offers the same actions for a selected server.
 
+The **Active** tab shows jobs verifying health first, then jobs executing work,
+waiting jobs, and queued jobs. File preparation, cleanup, and submitted commands
+awaiting host confirmation count as executing work. Within each group, older
+requests appear first, with recording order breaking timestamp ties. **History**
+shows the newest requests first. This display order does not change execution
+priority or capacity admission.
+
 **Create server** opens one dialog with game and map choices followed by automatic
 readiness checks. Blockers explain what needs attention before launch. Runtime
 and network overrides are under **Advanced settings**; image, configuration,

src/lib/provisioning-detail.ts

@@ -78,7 +78,7 @@ export interface ProvisioningDeployment {
     expected_map?: string | null;
 }
 export interface ProvisioningSource {
-    kind: 'campaign' | 'beta' | 'maintenance' | 'watchdog' | 'autoscaling';
+    kind: 'campaign' | 'beta' | 'maintenance' | 'watchdog' | 'autoscaling' | 'rebalance';
     source_id: string;
     server?: string;
     label: string;

src/lib/provisioning-list.ts

@@ -3,7 +3,7 @@ import type { ProvisioningDeployment } from './provisioning-detail.js';
 import type { ProvisioningSource } from './provisioning-detail.js';
 
 export const provisioningListStatuses = ['pending', 'running', 'awaiting-health', 'done', 'attention', 'failed', 'interrupted', 'warning', 'cancelled'] as const;
-export const provisioningListSources = ['manual', 'campaign', 'beta', 'maintenance', 'system'] as const;
+export const provisioningListSources = ['manual', 'campaign', 'beta', 'maintenance', 'rebalance', 'system'] as const;
 
 export type ProvisioningListRow = Pick<ProvisioningDeployment,
     'id' | 'server' | 'server_deleted_at' | 'action' | 'state' | 'stage' | 'started_at' |

src/lib/rebalancing.ts

@@ -0,0 +1,48 @@
+export interface RebalanceSettings {
+    hostIds: string[];
+    game: 'all' | 't5' | 't6';
+    maxMoves: number;
+}
+
+export interface RebalanceHostOption { id: string; name: string; eligible: boolean; reason: string | null }
+export interface RebalanceHostProjection {
+    id: string; name: string;
+    before: { cpuPercent: number; memoryPercent: number; instances: number };
+    after: { cpuPercent: number; memoryPercent: number; instances: number };
+}
+export interface RebalanceCandidate {
+    server: string; instanceId: string; generation: number;
+    sourceHostId: string; sourceHostName: string; targetHostId: string; targetHostName: string;
+    /** Estimated reduction in the source/destination pair's peak utilization, in percentage points. */
+    improvement: number;
+}
+export interface RebalancePlan {
+    hosts: RebalanceHostProjection[];
+    moves: RebalanceCandidate[];
+    exclusions: { server: string; reason: string }[];
+    warnings: string[];
+}
+export interface RebalancePreview extends RebalancePlan {
+    id: string; settings: RebalanceSettings; createdAt: string; expiresAt: string;
+}
+export interface RebalanceMove extends RebalanceCandidate {
+    id: string; state: 'planned' | 'moving' | 'done' | 'skipped' | 'failed';
+    operationId: string | null; message: string; updatedAt: string;
+}
+export interface RebalanceRun {
+    id: string; settings: RebalanceSettings;
+    state: 'running' | 'stopping' | 'done' | 'stopped' | 'failed';
+    createdAt: string; updatedAt: string; message: string; moves: RebalanceMove[];
+}
+export interface RebalanceReport { hosts: RebalanceHostOption[]; runs: RebalanceRun[] }
+
+export function rebalanceSettings(input: unknown): RebalanceSettings {
+    if (!input || typeof input !== 'object' || Array.isArray(input)) throw new Error('Provide rebalance settings.');
+    const value = input as Record<string, unknown>;
+    if (!Array.isArray(value.hostIds) || value.hostIds.length < 2 || value.hostIds.length > 100
+        || value.hostIds.some(id => typeof id !== 'string' || !id || id.length > 100)
+        || new Set(value.hostIds).size !== value.hostIds.length) throw new Error('Choose between 2 and 100 different hosts.');
+    if (!['all', 't5', 't6'].includes(value.game as string)) throw new Error('Choose a supported game.');
+    if (!Number.isInteger(value.maxMoves) || (value.maxMoves as number) < 1 || (value.maxMoves as number) > 20) throw new Error('Choose a move limit between 1 and 20.');
+    return { hostIds: [...value.hostIds as string[]].sort(), game: value.game as RebalanceSettings['game'], maxMoves: value.maxMoves as number };
+}

src/lib/server/backends/schema.ts

@@ -32,7 +32,7 @@ export function migrateNativeInstances(handle: Database.Database): void {
             deployment_id TEXT PRIMARY KEY REFERENCES server_deployments(id),
             source_host_id TEXT NOT NULL, instance_id TEXT NOT NULL REFERENCES native_instances(id),
             target_host_id TEXT, target_port INTEGER, target_bind TEXT,
-            remove_operation_id TEXT, removed_at TEXT
+            remove_operation_id TEXT, removed_at TEXT, runtime_password TEXT
         );
         CREATE INDEX IF NOT EXISTS ec_host_migrations_source ON ec_host_migrations(source_host_id);
         CREATE INDEX IF NOT EXISTS ec_host_migrations_target ON ec_host_migrations(target_host_id);
@@ -43,6 +43,9 @@ export function migrateNativeInstances(handle: Database.Database): void {
         );
         CREATE INDEX IF NOT EXISTS idx_ec_host_migrations_instance ON ec_host_migrations(instance_id);
     `);
+    if (!(handle.pragma('table_info(ec_host_migrations)') as { name: string }[]).some(column => column.name === 'runtime_password')) {
+        handle.exec('ALTER TABLE ec_host_migrations ADD COLUMN runtime_password TEXT');
+    }
     if (!handle.prepare("SELECT 1 FROM settings WHERE key='native_file_plans_v1'").get()) {
         handle.transaction(() => {
             handle.exec(`UPDATE native_provisioning SET files_json=(SELECT json_group_array(json_remove(value,'$.contentBase64'))

src/lib/server/db/index.ts

@@ -12,6 +12,7 @@ import { migrateServerHistory } from './server-history.js';
 import { migrateNativeInstances } from '../backends/schema.js';
 import { migrateServerKeys } from './server-keys.js';
 import { ensureAutoscalingSchema } from '../autoscaling/schema.js';
+import { ensureRebalancingSchema } from '../rebalancing/schema.js';
 import { installPerformanceIndexes } from './performance-indexes.js';
 import { migrateFileBlobs } from './file-blobs.js';
 import { enableIncrementalVacuum, retentionClocks } from './retention-clocks.js';
@@ -402,6 +403,7 @@ export function db(): Database.Database {
         migrationStep('native instance schema and file plans', () => migrateNativeInstances(handle));
         migrationStep('per-game server key pools', () => migrateServerKeys(handle));
         migrationStep('autoscaling schema', () => ensureAutoscalingSchema(handle));
+        migrationStep('rebalancing schema', () => ensureRebalancingSchema(handle));
         // File-blob migration removes the legacy content column altogether.
         // Reordering it first would unnecessarily copy every artifact twice.
         migrateFileBlobs(handle);

src/lib/server/hosts/availability.ts

@@ -3,6 +3,7 @@ import { fleetSettings } from '../settings.js';
 import { subscribe } from '../pipeline/bus.js';
 import { capacityCalibration } from './calibration.js';
 import { HEARTBEAT_MAX_AGE_MS, listHosts } from './index.js';
+import { committedRebalanceStartup } from '../provisioning/concurrency.js';
 import type { CapacityCalibration, Host, NativePlacement } from '../../hosts.js';
 import type { LaunchAvailability } from '../../deployment-plan.js';
 
@@ -37,9 +38,9 @@ function snapshotSchemaReady(handle: ReturnType<typeof db>): boolean {
 
 function queueSnapshot(handle: ReturnType<typeof db>, hosts: Map<string, HostAvailability>): Snapshot {
     const counts = handle.prepare(`SELECT
-        COALESCE(SUM(state IN ('running','awaiting-health')),0) AS active,
+        COALESCE(SUM(state IN ('running','awaiting-health') OR ${committedRebalanceStartup}),0) AS active,
         COALESCE(SUM(state='queued'),0) AS queued FROM server_deployments
-        WHERE action IN ('create','rebuild') AND state IN ('queued','running','awaiting-health')
+        WHERE action IN ('create','rebuild') AND (state IN ('queued','running','awaiting-health') OR ${committedRebalanceStartup})
         AND server_deleted_at IS NULL`).get() as { active: number; queued: number };
     return { hosts, ...counts, startupLimit: fleetSettings().createConcurrency, updatedAt: new Date().toISOString() };
 }

src/lib/server/hosts/capacity.ts

@@ -11,6 +11,7 @@ const CPU_USABLE_FRACTION = 0.95;
 export class CapacityError extends Error {}
 export interface PlacementOptions {
     hostId?: string; libraryId: string; image: string; bindAddress?: string;
+    excludeHostIds?: string[];
     thorough?: boolean; excludeInstanceId?: string;
     // Execution only: not-yet-submitted launches wait their turn instead of
     // requiring their entire backlog to fit before any one can start.
@@ -100,7 +101,7 @@ function projectedLoad(host: Host, options: PlacementOptions): number {
 export function choosePlacement(options: PlacementOptions): { host: Host; port: number; load: number } {
     const candidates: { host: Host; port: number; load: number }[] = [];
     const reasons: string[] = [];
-    for (const host of listHosts().filter(host => !options.hostId || host.id === options.hostId)) {
+    for (const host of listHosts().filter(host => (!options.hostId || host.id === options.hostId) && !options.excludeHostIds?.includes(host.id))) {
         try {
             requireReadyHost(host.id, { image: options.image, requiredBytes: INSTANCE_STORAGE_BYTES });
             assertReadyLibrary(host.id, options.libraryId);

src/lib/server/pipeline/worker.ts

@@ -1,6 +1,7 @@
 import { pruneHistory, pruneSampleHistory } from '../retention.js';
 import { collectFleetChat } from '../fleet/chat.js';
 import { runAutoscaling, pruneAutoscaling } from '../autoscaling/index.js';
+import { runRebalancing, pruneRebalancing } from '../rebalancing/index.js';
 import { config } from '../config.js';
 import { processProvisioning } from '../provisioning/worker.js';
 import { sweep } from './sweep.js';
@@ -59,9 +60,10 @@ function reconcileRequesters(): void {
         ['beta', 'Reconciling beta tests', betaTick],
         ['maintenance', 'Checking maintenance and recovery', runMaintenance],
         ['autoscaling', 'Evaluating lobby autoscaling', runAutoscaling],
+        ['rebalancing', 'Reconciling rebalance moves', runRebalancing],
         ['game-files', 'Removing unused game files', cleanupControllerLibraries],
         ['chat', 'Saving match chat', collectFleetChat],
-        ['retention', 'Cleaning expired history', async () => { pruneHistory(); pruneSampleHistory(); pruneAutoscaling(); }]
+        ['retention', 'Cleaning expired history', async () => { pruneHistory(); pruneSampleHistory(); pruneAutoscaling(); pruneRebalancing(); }]
     ];
     for (const [key, label, work] of requesters) {
         if (requesterFlights.has(key)) continue;
@@ -71,7 +73,7 @@ function reconcileRequesters(): void {
         }
         const flight = trackActivity(key, label, work)
             .catch(err => { console.error(`[worker] ${key} error:`, err); })
-            .then(async result => { if (key !== 'game-files' && key !== 'retention' && key !== 'chat' && (key !== 'autoscaling' || result === true)) await tick(true); })
+            .then(async result => { if (key !== 'game-files' && key !== 'retention' && key !== 'chat' && (!['autoscaling', 'rebalancing'].includes(key) || result === true)) await tick(true); })
             .finally(() => { requesterFlights.delete(key); });
         requesterFlights.set(key, flight);
     }

src/lib/server/provisioning/cancellation.ts

@@ -2,6 +2,15 @@ import { pauseScalingForOperation } from '../autoscaling/ownership.js';
 import { db } from '../db/index.js';
 import { ensureHostSchema } from '../hosts/schema.js';
 import { recordProgress } from './progress.js';
+import { requestRebalanceStopForOperation } from '../rebalancing/admission.js';
+
+export const rebalanceCancellationGuard = `NOT (d.triggered_by LIKE 'rebalance:%' AND EXISTS (
+    SELECT 1 FROM ec_host_migrations m WHERE m.deployment_id=d.id AND m.remove_operation_id IS NOT NULL))`;
+
+export function rebalanceCancellationBlocker(id: string): string | null {
+    return db().prepare(`SELECT 1 FROM server_deployments d WHERE d.id=? AND NOT (${rebalanceCancellationGuard})`).get(id)
+        ? 'This rebalance move has committed source removal and must finish recreating the lobby. Stop the rebalance run to withdraw its remaining moves.' : null;
+}
 
 export function provisioningCancelled(id: string): boolean {
     return !!db().prepare("SELECT 1 FROM server_deployments WHERE id=? AND state='cancelled'").get(id);
@@ -12,17 +21,20 @@ export function assertProvisioningNotCancelled(id: string): void {
 }
 
 export function canCancelProvisioningJob(id: string): boolean {
-    return !!db().prepare(`SELECT 1 FROM server_deployments WHERE id=? AND server_deleted_at IS NULL
-        AND state IN ('queued','waiting','running','awaiting-health','interrupted')`).get(id);
+    return !!db().prepare(`SELECT 1 FROM server_deployments d WHERE d.id=? AND d.server_deleted_at IS NULL
+        AND d.state IN ('queued','waiting','running','awaiting-health','interrupted') AND ${rebalanceCancellationGuard}`).get(id);
 }
 
 /** Cancel remaining work, not effects already performed by the host. */
-export function cancelProvisioningJob(id: string, reason = 'Cancelled by operator'): boolean {
+export function cancelProvisioningJob(id: string, reason = 'Cancelled by operator', options: { preserveRebalanceRun?: boolean } = {}): boolean {
     ensureHostSchema(db());
     return db().transaction(() => {
         if (provisioningCancelled(id)) return true;
         if (!canCancelProvisioningJob(id)) return false;
-        pauseScalingForOperation(id);
+        const rebalance = !!db().prepare("SELECT 1 FROM server_deployments WHERE id=? AND triggered_by LIKE 'rebalance:%'").get(id);
+        if (rebalance) {
+            if (!options.preserveRebalanceRun) requestRebalanceStopForOperation(id);
+        } else pauseScalingForOperation(id);
         const at = new Date().toISOString();
         // Only commands never delivered can be withdrawn. Delivered commands retain
         // their immutable IDs and receipts; replacements wait for their resolution.

src/lib/server/provisioning/concurrency.ts

@@ -4,6 +4,9 @@ import { fleetSettings } from '../settings.js';
 
 // A preparation owns capacity across asynchronous launch-file generation.
 export const preparingJobs = new Set<string>();
+/** A committed rebalance keeps its startup reservation while between source removal and creation. */
+export const committedRebalanceStartup = `(triggered_by LIKE 'rebalance:%' AND state IN ('waiting','interrupted')
+    AND EXISTS (SELECT 1 FROM ec_host_migrations m WHERE m.deployment_id=server_deployments.id AND m.remove_operation_id IS NOT NULL))`;
 interface Holder { id: string; action: string; host_id: string | null; target_host_id: string | null; removing: number }
 
 export function lifecycleHolders(): Holder[] {

src/lib/server/provisioning/history.ts

@@ -11,10 +11,12 @@ import { readProgress, recordProgress, type ProgressUpdate } from './progress.js
 import { bindNativeOperation } from './native.js';
 import { workerActivity } from '../pipeline/activity.js';
 import { canCancelCreation } from './cancel-creation.js';
-import { canCancelProvisioningJob, provisioningCancelled } from './cancellation.js';
+import { canCancelProvisioningJob, provisioningCancelled, rebalanceCancellationGuard } from './cancellation.js';
+import { committedRebalanceStartup } from './concurrency.js';
 import { nativeDeployment } from '../backends/native.js';
 import { safetyOverride, safetyOverrideAvailability } from './safety-override.js';
 import { serverTitle } from '../../ui/server-labels.js';
+import { provisioningQueueStatus } from '../../ui/provisioning-queue.js';
 import { provisioningPageSize, provisioningPageOffset, type ProvisioningListOptions, type ProvisioningListPage, type ProvisioningListRow } from '../../provisioning-list.js';
 
 export function beginDeployment(server: string, action: ProvisioningAction, triggeredBy = 'ui'): string {
@@ -22,7 +24,7 @@ export function beginDeployment(server: string, action: ProvisioningAction, trig
     db().transaction(() => {
         db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by, requested_by)
             VALUES (?, ?, ?, 'running', 'Starting', ?, ?, ?)`).run(id, server, action, new Date().toISOString(), triggeredBy,
-                triggeredBy === 'ui' ? currentRequestUsername() : null);
+                triggeredBy === 'ui' || triggeredBy.startsWith('rebalance:') ? currentRequestUsername() : null);
         bindNativeOperation(id, server);
         recordProgress(id, { phase: 'requested', observedAt: null, checkpoint: { key: 'requested', phase: 'requested',
             label: action === 'create' ? 'Server creation requested'
@@ -62,6 +64,11 @@ function warningMessages(raw: string): string[] {
 
 function sourceDetails(source: Omit<ProvisioningSource, 'label' | 'href'>): ProvisioningSource {
     const id = source.source_id;
+    if (source.kind === 'rebalance') {
+        const exists = !!db().prepare('SELECT 1 FROM rebalancing_runs WHERE id=?').get(id);
+        return { ...source, label: exists ? 'Rebalance run' : 'Rebalance history expired',
+            href: exists ? `/capacity?rebalance=1&rebalanceRun=${encodeURIComponent(id)}` : null };
+    }
     if (source.kind === 'autoscaling') {
         const exists = !!db().prepare('SELECT 1 FROM autoscaling_policies WHERE id=?').get(id);
         return { ...source, label: exists ? 'Autoscaling policy' : 'Deleted autoscaling policy', href: exists ? `/autoscaling#${encodeURIComponent(id)}` : null };
@@ -93,7 +100,7 @@ function capacityFor(deployment: ProvisioningDeployment, evaluatedAt: string): P
     const pools: ProvisioningCapacity['pools'] = [];
     if (deployment.action === 'create' || deployment.action === 'rebuild') {
         const jobs = db().prepare(`SELECT id, server, state FROM server_deployments
-            WHERE server_deleted_at IS NULL AND action IN ('create','rebuild') AND state IN ('queued','running','awaiting-health')
+            WHERE server_deleted_at IS NULL AND action IN ('create','rebuild') AND (state IN ('queued','running','awaiting-health') OR ${committedRebalanceStartup})
             ORDER BY started_at, rowid`).all() as { id: string; server: string; state: string }[];
         const holders = jobs.filter(job => job.state !== 'queued').map(({ id, server }) => ({ id, server }));
         const order = jobs.filter(job => job.state === 'queued').findIndex(job => job.id === deployment.id);
@@ -126,6 +133,7 @@ export function getDeployment(id: string, options: { events?: 'all' | 'recent' }
         const { observation_json: _snapshot, phase_key: _phase, phase_started_at: _phaseStarted, observed_at: _observed, ...deployment } = row;
         const sources = (db().prepare('SELECT kind, source_id, server FROM provisioning_sources WHERE operation_id = ? ORDER BY kind, source_id')
             .all(id) as Omit<ProvisioningSource, 'label' | 'href'>[]).map(sourceDetails);
+        if (row.triggered_by.startsWith('rebalance:')) sources.push(sourceDetails({ kind: 'rebalance', source_id: row.triggered_by.slice('rebalance:'.length) }));
         const launch = recipeSummaryForDeployment(id);
         const collection = db().prepare(`SELECT c.id, c.name FROM (
             SELECT collection_id, deployment_id FROM server_collection_members
@@ -270,7 +278,8 @@ const deploymentListColumns = `d.id, d.server, d.server_deleted_at, d.action, d.
         CASE WHEN json_valid(d.observation_json) THEN json_extract(d.observation_json,'$.blocker.code') END AS waitCode,
         (d.server_deleted_at IS NULL AND EXISTS(SELECT 1 FROM servers s WHERE s.name = d.server AND s.destroyed_at IS NULL)) AS server_exists,
         (d.server_deleted_at IS NULL AND d.action = 'destroy' AND d.state = 'waiting' AND d.stage != ?) AS canCancelRemoval,
-        (d.server_deleted_at IS NULL AND d.state IN ('queued','waiting','running','awaiting-health','interrupted')) AS canCancel
+        (d.server_deleted_at IS NULL AND d.state IN ('queued','waiting','running','awaiting-health','interrupted')
+            AND ${rebalanceCancellationGuard}) AS canCancel
     `;
 
 function deploymentListQuery(options: ProvisioningListOptions) {
@@ -285,8 +294,9 @@ function deploymentListQuery(options: ProvisioningListOptions) {
         OR ${kinds.map(kind => `d.triggered_by LIKE '${kind}:%'`).join(' OR ')})`;
     if (options.source === 'campaign' || options.source === 'beta') conditions.push(linked([options.source]));
     else if (options.source === 'maintenance') conditions.push(linked(['maintenance', 'watchdog']));
+    else if (options.source === 'rebalance') conditions.push("d.triggered_by LIKE 'rebalance:%'");
     else if (options.source === 'manual' || options.source === 'system') {
-        conditions.push(`NOT ${linked(['campaign', 'beta', 'maintenance', 'watchdog'])}`);
+        conditions.push(`NOT ${linked(['campaign', 'beta', 'maintenance', 'watchdog', 'rebalance'])}`);
         conditions.push(options.source === 'manual' ? "d.triggered_by='ui'" : "d.triggered_by!='ui'");
     }
     const search = (options.server ?? '').slice(0, 100).toLowerCase();
@@ -301,11 +311,28 @@ function deploymentListQuery(options: ProvisioningListOptions) {
         limit: provisioningPageSize(options.limit ?? 25), offset: provisioningPageOffset(options.offset ?? 0) };
 }
 
+function activeDeploymentRank(row: ProvisioningListRow): number {
+    const { group } = provisioningQueueStatus(row);
+    if (group === 'verifying') return 0;
+    if (group === 'running') return 1;
+    // Launch preparation keeps its queued/waiting state until command submission.
+    if (row.phase === 'prepare' && !row.waitCode) return 1;
+    // Remote file staging currently records its progress only in the stage message.
+    if (row.waitCode === 'native-host' && ['Staging and verifying files on the host.',
+        'Waiting for verified remote file staging.'].includes(row.stage)) return 1;
+    if (group === 'queued') return 3;
+    // A submitted command remains in progress while its agent result is pending.
+    if (row.state === 'running' && row.phase === 'container-action' && row.waitCode === 'native-host') return 1;
+    return 2;
+}
+
 function deploymentListRows(query: ReturnType<typeof deploymentListQuery>, active: boolean): ProvisioningListRow[] {
     const rows = db().prepare(`SELECT ${deploymentListColumns} FROM server_deployments d WHERE ${query.where}
-        ORDER BY d.started_at DESC, d.rowid DESC ${active ? '' : 'LIMIT ? OFFSET ?'}`)
+        ORDER BY ${active ? 'd.started_at ASC, d.rowid ASC' : 'd.started_at DESC, d.rowid DESC LIMIT ? OFFSET ?'}`)
         .all(REMOVING_SERVER, ...query.parameters, ...(active ? [] : [query.limit, query.offset])) as ProvisioningListRow[];
     if (!rows.length) return rows;
+    // Stable sorting keeps request/insertion order within each progress group.
+    if (active) rows.sort((left, right) => activeDeploymentRank(left) - activeDeploymentRank(right));
     const links = db().prepare(`SELECT operation_id,kind,source_id FROM provisioning_sources
         WHERE operation_id IN (SELECT value FROM json_each(?)) ORDER BY kind,source_id`).all(JSON.stringify(rows.map(row => row.id))) as
         (Omit<ProvisioningSource, 'label' | 'href'> & { operation_id: string })[];
@@ -314,7 +341,7 @@ function deploymentListRows(query: ReturnType<typeof deploymentListQuery>, activ
     const cache = new Map<string, ProvisioningSource>();
     for (const row of rows) {
         const sources = byOperation.get(row.id) ?? [];
-        const fallback = row.triggered_by.match(/^(campaign|beta|maintenance|watchdog|autoscaling):(.+)$/);
+        const fallback = row.triggered_by.match(/^(campaign|beta|maintenance|watchdog|autoscaling|rebalance):(.+)$/);
         if (fallback && !sources.some(source => source.kind === fallback[1] && source.source_id === fallback[2])) sources.unshift({ kind: fallback[1] as ProvisioningSource['kind'], source_id: fallback[2] });
         row.sources = sources.map(source => {
             const key = `${source.kind}:${source.source_id}`;

src/lib/server/provisioning/host-drain.ts

@@ -1,29 +1,13 @@
-import { concurrencyBlocker } from './concurrency.js';
 import { db } from '../db/index.js';
-import { provisioningCancelled } from './cancellation.js';
-import { nativeInstance, nativeRequest, type NativeInstance } from '../backends/native.js';
-import { getHost, getHostOperationStatus, listHosts, queueHostOperation, reserveHostPort, updateHost } from '../hosts/index.js';
-import { choosePlacement } from '../hosts/capacity.js';
-import { scheduleAvailabilityRefresh } from '../hosts/availability.js';
+import type { NativeInstance } from '../backends/native.js';
+import { listHosts, updateHost } from '../hosts/index.js';
 import { activeBetaRemovalBlocker, getServer } from '../fleet/servers.js';
-import { getServerKey } from '../fleet/server-keys.js';
-import { captureRecipePlan, recipeForDeployment, recipeStorage, type LaunchRecipePlan } from '../fleet/launch-recipes.js';
-import { probePlayerEvidence } from '../fleet/status-evidence.js';
+import { captureRecipePlan, recipeStorage } from '../fleet/launch-recipes.js';
 import { beginDeployment, deploymentStage } from './history.js';
-import { fleetSettings } from '../settings.js';
 import type { NativePlacement } from '../../hosts.js';
 
-interface Migration {
-    deployment_id: string; source_host_id: string; instance_id: string;
-    target_host_id: string | null; target_port: number | null; target_bind: string | null;
-    remove_operation_id: string | null; removed_at: string | null;
-}
-interface Job { id: string; server: string; action: string; state: string; stage: string }
-type Recipe = LaunchRecipePlan & { native: NativePlacement };
-
-export function hostMigration(id: string): Migration | undefined {
-    return db().prepare('SELECT * FROM ec_host_migrations WHERE deployment_id=?').get(id) as Migration | undefined;
-}
+export { captureMoveRecipe } from './move-recipes.js';
+export { hostMigration, isRebalanceMigration, processMigrationRemoval, requestInstanceMove } from './migrations.js';
 
 /** Drain requests survive controller restarts and never make a source slot available early. */
 export function reconcileHostDrains(): void {
@@ -47,7 +31,8 @@ export function reconcileHostDrains(): void {
                 if (!server?.map_slug || activeBetaRemovalBlocker(instance.server)) continue;
                 // An explicit cancellation withdraws this instance's current drain attempt.
                 const latest = db().prepare(`SELECT d.state FROM ec_host_migrations m JOIN server_deployments d ON d.id=m.deployment_id
-                    WHERE m.instance_id=? AND d.server_deleted_at IS NULL ORDER BY d.rowid DESC LIMIT 1`).get(instance.id) as { state: string } | undefined;
+                    WHERE m.instance_id=? AND m.source_host_id=? AND d.triggered_by='host-drain'
+                    AND d.server_deleted_at IS NULL ORDER BY d.rowid DESC LIMIT 1`).get(instance.id, host.id) as { state: string } | undefined;
                 if (latest?.state === 'cancelled') continue;
                 if (db().prepare(`SELECT 1 FROM server_deployments WHERE server=? AND server_deleted_at IS NULL
                     AND state IN ('queued','waiting','running','awaiting-health','interrupted')`).get(instance.server)) continue;
@@ -69,94 +54,3 @@ export function reconcileHostDrains(): void {
         })();
     }
 }
-
-/** Reserve a destination, remove the source once, then hand creation to the normal health-verified rebuild queue. */
-export async function processMigrationRemoval(job: Job, instance: NativeInstance, migration: Migration,
-    eligibility: () => string | null): Promise<void> {
-    if (provisioningCancelled(job.id)) return;
-    const hold = (message: string) => { if (!provisioningCancelled(job.id)) deploymentStage(job.id, message, { blocker: { code: 'native-host', message, automatic: true } }); };
-    if (migration.remove_operation_id) {
-        const operation = getHostOperationStatus(migration.remove_operation_id);
-        if (operation?.state === 'failed' && Date.now() - Date.parse(operation.updatedAt) >= 60_000) {
-            // A failed receipt certifies no mutation began. Unknown results keep
-            // their reservation and original operation ID until reconciled.
-            db().transaction(() => {
-                db().prepare('DELETE FROM ec_host_ports WHERE host_id=? AND instance_id=?').run(migration.target_host_id, instance.id);
-                db().prepare(`UPDATE ec_host_migrations SET target_host_id=NULL,target_port=NULL,target_bind=NULL,
-                    remove_operation_id=NULL WHERE deployment_id=?`).run(job.id);
-                db().prepare("UPDATE server_deployments SET state='waiting',submitted_at=NULL WHERE id=?").run(job.id);
-                scheduleAvailabilityRefresh();
-            })();
-            hold('Source removal failed before mutation; checking destination capacity again before retrying.');
-            return;
-        }
-        if (operation?.state !== 'completed') {
-            hold(operation?.error ?? 'Waiting for confirmed source removal. The original agent operation will reconcile.');
-            return;
-        }
-        db().transaction(() => {
-            if (hostMigration(job.id)?.removed_at) return;
-            const current = nativeInstance(job.server);
-            if (current?.id !== instance.id || current.generation !== instance.generation || current.host_id !== migration.source_host_id) throw new Error('Migration source changed.');
-            const target = getHost(migration.target_host_id!);
-            if (!target) throw new Error('Migration destination is unavailable.');
-            const recipe = recipeForDeployment(job.id)!.recipe as Recipe;
-            recipe.native = { libraryId: instance.library_id, image: instance.image, hostId: target.id,
-                bindAddress: migration.target_bind!, publicAddress: target.publicAddress };
-            const stored = recipeStorage(recipe);
-            db().prepare('UPDATE server_launch_recipes SET recipe=?,recipe_summary=? WHERE deployment_id=?').run(stored.recipe, stored.recipeSummary, job.id);
-            db().prepare('DELETE FROM ec_host_ports WHERE host_id=? AND instance_id=?').run(instance.host_id, instance.id);
-            db().prepare(`UPDATE native_instances SET host_id=?,generation=generation+1,bind_address=?,port=?,public_address=?,public_port=NULL WHERE id=?`)
-                .run(target.id, migration.target_bind, migration.target_port, target.publicAddress, instance.id);
-            db().prepare('UPDATE native_provisioning SET generation=generation+1 WHERE deployment_id=?').run(job.id);
-            db().prepare('UPDATE server_creation_jobs SET port=? WHERE deployment_id=?').run(migration.target_port, job.id);
-            db().prepare(`UPDATE servers SET game_port=?,container_state='starting',container_checked_at=NULL,
-                last_players=NULL,last_roster=NULL,last_seen_at=NULL,last_error=NULL,rcon_failures=0,active_since=NULL,inactive_since=NULL,
-                game_started_at=NULL,game_ended_at=NULL WHERE name=?`).run(migration.target_port, job.server);
-            db().prepare('UPDATE ec_host_migrations SET removed_at=? WHERE deployment_id=?').run(operation.updatedAt, job.id);
-            db().prepare("UPDATE server_deployments SET state='waiting',submitted_at=NULL WHERE id=?").run(job.id);
-            deploymentStage(job.id, `Source removed; rebuilding on ${target.name}`, { phase: 'startup-queue', blocker: null });
-            scheduleAvailabilityRefresh();
-        })();
-        return;
-    }
-    if (!getHost(instance.host_id)?.connected) { hold('Waiting for the source host to reconnect.'); return; }
-    const reason = eligibility() ?? activeBetaRemovalBlocker(job.server);
-    if (reason) { hold(reason); return; }
-    if (!getServerKey(job.server)) { hold('Save a server key before migrating this instance.'); return; }
-    const options = { libraryId: instance.library_id, image: instance.image, thorough: true, dispatch: true };
-    // No source mutation is submitted unless another host can take this instance.
-    choosePlacement(options);
-    const health = await nativeRequest<{ state: string }>(instance, 'instance.inspect');
-    if (provisioningCancelled(job.id)) return;
-    if (!['stopped','exited','created','dead','missing'].includes(health.state)) {
-        const occupancy = await probePlayerEvidence(job.server);
-        if (provisioningCancelled(job.id)) return;
-        if (occupancy.players !== 0) { hold(occupancy.players === null ? 'Waiting for a fresh local player count.' : `Waiting for ${occupancy.players} player(s) to leave.`); return; }
-    }
-    db().transaction(() => {
-        const state = db().prepare('SELECT state FROM server_deployments WHERE id=? AND server_deleted_at IS NULL').get(job.id) as { state: string } | undefined;
-        if (!state || !['waiting','queued','interrupted'].includes(state.state) || hostMigration(job.id)?.remove_operation_id) return;
-        const current = nativeInstance(job.server);
-        if (current?.id !== instance.id || current.generation !== instance.generation || getHost(instance.host_id)?.mode !== 'draining') return;
-        const blocker = eligibility() ?? activeBetaRemovalBlocker(job.server);
-        if (blocker) { hold(blocker); return; }
-        const removing = db().prepare("SELECT COUNT(*) AS n FROM ec_host_operations WHERE kind='instance.remove' AND state IN ('pending','accepted','running','unknown')").get() as { n: number };
-        if (removing.n >= fleetSettings().destroyConcurrency) { hold('Waiting for destruction capacity.'); return; }
-        const { host, port } = choosePlacement(options);
-        const capacityBlocker = concurrencyBlocker(job.id, 'rebuild', instance.host_id, true) ?? concurrencyBlocker(job.id, 'rebuild', host.id);
-        if (capacityBlocker) { hold(capacityBlocker); return; }
-        reserveHostPort(host.id, instance.id, { port, bindAddress: host.bindAddress, protocol: 'udp' });
-        reserveHostPort(host.id, instance.id, { port, bindAddress: host.bindAddress, protocol: 'tcp' });
-        const attempt = (db().prepare(`SELECT COUNT(*) AS n FROM ec_host_operations
-            WHERE host_id=? AND instance_id=? AND kind='instance.remove' AND idempotency_key LIKE ?`)
-            .get(instance.host_id, instance.id, `${job.id}:migrate-remove:%`) as { n: number }).n;
-        const operation = queueHostOperation(instance.host_id, 'instance.remove', {}, {
-            instanceId: instance.id, generation: instance.generation, idempotencyKey: `${job.id}:migrate-remove:${attempt}`
-        });
-        db().prepare('UPDATE ec_host_migrations SET target_host_id=?,target_port=?,target_bind=?,remove_operation_id=? WHERE deployment_id=?')
-            .run(host.id, port, host.bindAddress, operation.id, job.id);
-        db().prepare("UPDATE server_deployments SET state='running',submitted_at=? WHERE id=?").run(new Date().toISOString(), job.id);
-        deploymentStage(job.id, `Capacity reserved on ${host.name}; waiting for source removal`, { phase: 'container-action', blocker: null });
-    })();
-}

src/lib/server/provisioning/launch-preparation.ts

@@ -15,6 +15,8 @@ export interface NativeLaunchInput {
     rconPassword: string;
     rebuild: boolean;
     safetyOverride?: boolean;
+    /** Rebalance replays recorded deployed bytes instead of current promoted files. */
+    frozenRecipeId?: string;
 }
 
 export interface PreparedNativeLaunch {

src/lib/server/provisioning/launch-worker.ts

@@ -2,7 +2,7 @@ import { parentPort } from 'node:worker_threads';
 import { posix } from 'node:path';
 import { db } from '../db/index.js';
 import { relDestForKind } from '../config.js';
-import { captureRecipe, summarizeRecipe } from '../fleet/launch-recipes.js';
+import { captureRecipe, recipeForDeployment, summarizeRecipe, type LaunchRecipe } from '../fleet/launch-recipes.js';
 import { sha256Buffer } from '../fleet/files.js';
 import type { NativeLaunchInput } from './launch-preparation.js';
 import { choosePlacement, type PlacementOptions } from '../hosts/capacity.js';
@@ -17,12 +17,26 @@ parentPort!.on('message', (message: { id: number; kind: 'launch'; input: NativeL
         }
         const input = message.input;
         // All file versions and composed configuration belong to one snapshot.
-        const recipe = db().transaction(() => ({ ...captureRecipe(input.game, input.mapSlug, input.rconPassword),
-            native: input.native, image: input.instance.image }))();
+        const { recipe, password } = db().transaction(() => {
+            if (!input.frozenRecipeId) return { recipe: { ...captureRecipe(input.game, input.mapSlug, input.rconPassword),
+                native: input.native, image: input.instance.image }, password: input.rconPassword };
+            const stored = recipeForDeployment(input.frozenRecipeId)?.recipe;
+            const migration = db().prepare('SELECT runtime_password FROM ec_host_migrations WHERE deployment_id=?')
+                .get(input.frozenRecipeId) as { runtime_password: string | null } | undefined;
+            if (!stored || !migration?.runtime_password || stored.files.some(file => file.content === undefined)) {
+                throw new Error('The preserved deployed recipe or runtime credentials are unavailable.');
+            }
+            for (const file of stored.files) {
+                const bytes = Buffer.from(file.content!, 'base64');
+                if (bytes.length !== file.size || sha256Buffer(bytes) !== file.sha256) throw new Error('Preserved deployed file verification failed.');
+            }
+            return { recipe: { ...stored, native: input.native, image: input.instance.image } as LaunchRecipe,
+                password: migration.runtime_password };
+        })();
         const instance = input.instance;
         const payload = {
             name: instance.server, game: recipe.game, libraryId: instance.library_id, image: instance.image,
-            port: instance.port, bindAddress: instance.bind_address, rconPassword: input.rconPassword,
+            port: instance.port, bindAddress: instance.bind_address, rconPassword: password,
             env: { PLUTAINER_SERVER_NAME: instance.server, PLUTAINER_PORT: String(instance.port),
                 PLUTAINER_GAME: recipe.game === 't6' ? 't6zm' : 't5sp', PLUTAINER_CONFIG_FILE: recipe.configFile,
                 PLUTAINER_MAP_ROTATE: 'false',

src/lib/server/provisioning/migrations.ts

@@ -0,0 +1,183 @@
+import { concurrencyBlocker, preparingJobs, committedRebalanceStartup } from './concurrency.js';
+import { db } from '../db/index.js';
+import { provisioningCancelled } from './cancellation.js';
+import { nativeInstance, nativeRequest, runtimePasswordForServer, type NativeInstance } from '../backends/native.js';
+import { getHost, getHostOperationStatus, queueHostOperation, reserveHostPort } from '../hosts/index.js';
+import { choosePlacement } from '../hosts/capacity.js';
+import { scheduleAvailabilityRefresh } from '../hosts/availability.js';
+import { activeBetaRemovalBlocker, getServer } from '../fleet/servers.js';
+import { getServerKey } from '../fleet/server-keys.js';
+import { recipeForDeployment, recipeStorage, recipeSummaryForDeployment, type LaunchRecipePlan } from '../fleet/launch-recipes.js';
+import { captureMoveRecipe, moveRecipeMatches, moveRecipeStorage } from './move-recipes.js';
+import { rebalanceAdmission } from '../rebalancing/admission.js';
+import { probePlayerEvidence } from '../fleet/status-evidence.js';
+import { beginDeployment, deploymentStage } from './history.js';
+import { fleetSettings } from '../settings.js';
+import type { NativePlacement } from '../../hosts.js';
+
+interface Migration {
+    deployment_id: string; source_host_id: string; instance_id: string;
+    target_host_id: string | null; target_port: number | null; target_bind: string | null;
+    remove_operation_id: string | null; removed_at: string | null;
+}
+interface Job { id: string; server: string; action: string; state: string; stage: string }
+type Recipe = LaunchRecipePlan & { native: NativePlacement };
+export { captureMoveRecipe } from './move-recipes.js';
+
+export function isRebalanceMigration(id: string): boolean {
+    return !!db().prepare("SELECT 1 FROM server_deployments WHERE id=? AND triggered_by LIKE 'rebalance:%'").get(id);
+}
+
+/** Queue a move to one explicit host; its source remains active throughout. */
+export function requestInstanceMove(serverName: string, targetHostId: string,
+    options: { triggeredBy: string; recipe?: LaunchRecipePlan }): string {
+    return db().transaction(() => {
+        const instance = nativeInstance(serverName);
+        const server = getServer(serverName);
+        if (!instance || !server?.map_slug) throw new Error('The native instance is unavailable.');
+        if (instance.host_id === targetHostId) throw new Error('Choose a different destination host.');
+        if (getHost(instance.host_id)?.mode !== 'active') throw new Error('The source host must be active.');
+        if (!options.triggeredBy.startsWith('rebalance:')) throw new Error('An instance move requires a rebalance run.');
+        const blocker = activeBetaRemovalBlocker(serverName);
+        if (blocker) throw new Error(blocker);
+        if (!getServerKey(serverName)) throw new Error('Save a server key before moving this instance.');
+        if (db().prepare(`SELECT 1 FROM server_deployments WHERE server=? AND server_deleted_at IS NULL
+            AND state IN ('queued','waiting','running','awaiting-health','interrupted')`).get(serverName)) {
+            throw new Error('Another provisioning operation is active for this instance.');
+        }
+        const recipe = options.recipe ?? captureMoveRecipe(serverName);
+        if (recipe.game !== server.game || recipe.mapSlug !== server.map_slug || recipe.image !== instance.image
+            || !moveRecipeMatches(serverName, recipe)) throw new Error('The deployed files changed; generate a fresh rebalance preview.');
+        const password = runtimePasswordForServer(serverName);
+        if (!password) throw new Error('The deployed runtime credentials are unavailable.');
+        choosePlacement({ hostId: targetHostId, excludeHostIds: [instance.host_id], libraryId: instance.library_id,
+            image: instance.image, thorough: true, dispatch: true });
+        const id = beginDeployment(serverName, 'rebuild', options.triggeredBy);
+        db().prepare("UPDATE server_deployments SET state='waiting' WHERE id=?").run(id);
+        db().prepare('INSERT INTO server_creation_jobs(deployment_id,server,port,game_mode,map_slug,expected_map) VALUES(?,?,NULL,?,?,?)')
+            .run(id, serverName, 'zombie', recipe.mapSlug, recipe.expectedMap);
+        const stored = moveRecipeStorage({ ...recipe, native: { libraryId: instance.library_id, image: instance.image, hostId: targetHostId } });
+        db().prepare('INSERT INTO server_launch_recipes(deployment_id,recipe,recipe_summary) VALUES(?,?,?)').run(id, stored.recipe, stored.recipeSummary);
+        db().prepare('INSERT INTO ec_host_migrations(deployment_id,source_host_id,instance_id,runtime_password) VALUES(?,?,?,?)')
+            .run(id, instance.host_id, instance.id, password);
+        deploymentStage(id, 'Rebalance queued; waiting to confirm an empty lobby and destination capacity', { phase: 'eligibility', recipeStatus: 'queued' });
+        return id;
+    })();
+}
+
+export function hostMigration(id: string): Migration | undefined {
+    return db().prepare('SELECT * FROM ec_host_migrations WHERE deployment_id=?').get(id) as Migration | undefined;
+}
+
+/** Reserve a destination, remove the source once, then hand creation to the normal health-verified rebuild queue. */
+export async function processMigrationRemoval(job: Job, instance: NativeInstance, migration: Migration,
+    eligibility: () => string | null): Promise<void> {
+    if (provisioningCancelled(job.id)) return;
+    const hold = (message: string) => { if (!provisioningCancelled(job.id)) deploymentStage(job.id, message, { blocker: { code: 'native-host', message, automatic: true } }); };
+    if (migration.remove_operation_id) {
+        const operation = getHostOperationStatus(migration.remove_operation_id);
+        if (operation?.state === 'failed' && Date.now() - Date.parse(operation.updatedAt) >= 60_000) {
+            // A failed receipt certifies no mutation began. Unknown results keep
+            // their reservation and original operation ID until reconciled.
+            db().transaction(() => {
+                db().prepare('DELETE FROM ec_host_ports WHERE host_id=? AND instance_id=?').run(migration.target_host_id, instance.id);
+                db().prepare(`UPDATE ec_host_migrations SET target_host_id=NULL,target_port=NULL,target_bind=NULL,
+                    remove_operation_id=NULL WHERE deployment_id=?`).run(job.id);
+                db().prepare("UPDATE server_deployments SET state='waiting',submitted_at=NULL WHERE id=?").run(job.id);
+                scheduleAvailabilityRefresh();
+            })();
+            hold('Source removal failed before mutation; checking destination capacity again before retrying.');
+            return;
+        }
+        if (operation?.state !== 'completed') {
+            hold(operation?.error ?? 'Waiting for confirmed source removal. The original agent operation will reconcile.');
+            return;
+        }
+        db().transaction(() => {
+            if (hostMigration(job.id)?.removed_at) return;
+            const current = nativeInstance(job.server);
+            if (current?.id !== instance.id || current.generation !== instance.generation || current.host_id !== migration.source_host_id) throw new Error('Migration source changed.');
+            const target = getHost(migration.target_host_id!);
+            if (!target) throw new Error('Migration destination is unavailable.');
+            const rebalance = isRebalanceMigration(job.id);
+            const recipe = (rebalance ? { format: 1, ...recipeSummaryForDeployment(job.id)!.recipe }
+                : recipeForDeployment(job.id)!.recipe) as Recipe;
+            recipe.native = { libraryId: instance.library_id, image: instance.image, hostId: target.id,
+                bindAddress: migration.target_bind!, publicAddress: target.publicAddress };
+            const stored = rebalance ? moveRecipeStorage(recipe) : recipeStorage(recipe);
+            db().prepare('UPDATE server_launch_recipes SET recipe=?,recipe_summary=? WHERE deployment_id=?').run(stored.recipe, stored.recipeSummary, job.id);
+            db().prepare('DELETE FROM ec_host_ports WHERE host_id=? AND instance_id=?').run(instance.host_id, instance.id);
+            db().prepare(`UPDATE native_instances SET host_id=?,generation=generation+1,bind_address=?,port=?,public_address=?,public_port=NULL WHERE id=?`)
+                .run(target.id, migration.target_bind, migration.target_port, target.publicAddress, instance.id);
+            db().prepare('UPDATE native_provisioning SET generation=generation+1 WHERE deployment_id=?').run(job.id);
+            db().prepare('UPDATE server_creation_jobs SET port=? WHERE deployment_id=?').run(migration.target_port, job.id);
+            db().prepare(`UPDATE servers SET game_port=?,container_state='starting',container_checked_at=NULL,
+                last_players=NULL,last_roster=NULL,last_seen_at=NULL,last_error=NULL,rcon_failures=0,active_since=NULL,inactive_since=NULL,
+                game_started_at=NULL,game_ended_at=NULL WHERE name=?`).run(migration.target_port, job.server);
+            db().prepare('UPDATE ec_host_migrations SET removed_at=? WHERE deployment_id=?').run(operation.updatedAt, job.id);
+            db().prepare("UPDATE server_deployments SET state='waiting',submitted_at=NULL WHERE id=?").run(job.id);
+            deploymentStage(job.id, `Source removed; rebuilding on ${target.name}`, { phase: 'startup-queue', blocker: null });
+            scheduleAvailabilityRefresh();
+        })();
+        return;
+    }
+    if (!getHost(instance.host_id)?.connected) { hold('Waiting for the source host to reconnect.'); return; }
+    const reason = rebalanceAdmission(job.id) ?? eligibility() ?? activeBetaRemovalBlocker(job.server);
+    if (reason) { hold(reason); return; }
+    if (!getServerKey(job.server)) { hold('Save a server key before migrating this instance.'); return; }
+    const rebalance = isRebalanceMigration(job.id);
+    const recipe = rebalance ? recipeSummaryForDeployment(job.id)!.recipe : null;
+    if (rebalance && !moveRecipeMatches(job.server, { format: 1, ...recipe! })) {
+        hold('The deployed files changed; stop this run and generate a fresh rebalance preview.'); return;
+    }
+    const options = { libraryId: instance.library_id, image: instance.image, thorough: true, dispatch: true,
+        excludeHostIds: [instance.host_id], ...(rebalance ? { hostId: recipe!.native!.hostId } : {}) };
+    // No source mutation is submitted unless another host can take this instance.
+    choosePlacement(options);
+    const health = await nativeRequest<{ state: string; healthy?: boolean }>(instance, 'instance.inspect');
+    if (provisioningCancelled(job.id)) return;
+    if (rebalance && (health.state !== 'running' || health.healthy !== true)) {
+        hold('The source lobby is no longer healthy and running. Waiting for current health evidence.'); return;
+    }
+    if (!['stopped','exited','created','dead','missing'].includes(health.state)) {
+        const occupancy = await probePlayerEvidence(job.server);
+        if (provisioningCancelled(job.id)) return;
+        if (occupancy.players !== 0) { hold(occupancy.players === null ? 'Waiting for a fresh local player count.' : `Waiting for ${occupancy.players} player(s) to leave.`); return; }
+    }
+    db().transaction(() => {
+        const state = db().prepare('SELECT state FROM server_deployments WHERE id=? AND server_deleted_at IS NULL').get(job.id) as { state: string } | undefined;
+        if (!state || !['waiting','queued','interrupted'].includes(state.state) || hostMigration(job.id)?.remove_operation_id) return;
+        const current = nativeInstance(job.server);
+        if (current?.id !== instance.id || current.generation !== instance.generation
+            || getHost(instance.host_id)?.mode !== (rebalance ? 'active' : 'draining')) return;
+        const blocker = rebalanceAdmission(job.id) ?? eligibility() ?? activeBetaRemovalBlocker(job.server);
+        if (blocker) { hold(blocker); return; }
+        if (rebalance && !moveRecipeMatches(job.server, { format: 1, ...recipe! })) {
+            hold('The deployed files changed; stop this run and generate a fresh rebalance preview.'); return;
+        }
+        if (rebalance) {
+            const startups = db().prepare(`SELECT COUNT(*) AS n FROM (
+                SELECT id FROM server_deployments WHERE id!=? AND server_deleted_at IS NULL
+                    AND action IN ('create','rebuild') AND (state IN ('running','awaiting-health') OR ${committedRebalanceStartup})
+                UNION SELECT value FROM json_each(?) WHERE value!=?)`).get(job.id, JSON.stringify([...preparingJobs]), job.id) as { n: number };
+            if (startups.n >= fleetSettings().createConcurrency) { hold('Waiting for startup capacity before moving this lobby.'); return; }
+        }
+        const removing = db().prepare("SELECT COUNT(*) AS n FROM ec_host_operations WHERE kind='instance.remove' AND state IN ('pending','accepted','running','unknown')").get() as { n: number };
+        if (removing.n >= fleetSettings().destroyConcurrency) { hold('Waiting for destruction capacity.'); return; }
+        const { host, port } = choosePlacement(options);
+        const capacityBlocker = concurrencyBlocker(job.id, 'rebuild', instance.host_id, true) ?? concurrencyBlocker(job.id, 'rebuild', host.id);
+        if (capacityBlocker) { hold(capacityBlocker); return; }
+        reserveHostPort(host.id, instance.id, { port, bindAddress: host.bindAddress, protocol: 'udp' });
+        reserveHostPort(host.id, instance.id, { port, bindAddress: host.bindAddress, protocol: 'tcp' });
+        const attempt = (db().prepare(`SELECT COUNT(*) AS n FROM ec_host_operations
+            WHERE host_id=? AND instance_id=? AND kind='instance.remove' AND idempotency_key LIKE ?`)
+            .get(instance.host_id, instance.id, `${job.id}:migrate-remove:%`) as { n: number }).n;
+        const operation = queueHostOperation(instance.host_id, 'instance.remove', {}, {
+            instanceId: instance.id, generation: instance.generation, idempotencyKey: `${job.id}:migrate-remove:${attempt}`
+        });
+        db().prepare('UPDATE ec_host_migrations SET target_host_id=?,target_port=?,target_bind=?,remove_operation_id=? WHERE deployment_id=?')
+            .run(host.id, port, host.bindAddress, operation.id, job.id);
+        db().prepare("UPDATE server_deployments SET state='running',submitted_at=? WHERE id=?").run(new Date().toISOString(), job.id);
+        deploymentStage(job.id, `Capacity reserved on ${host.name}; waiting for source removal`, { phase: 'container-action', blocker: null });
+    })();
+}

src/lib/server/provisioning/move-recipes.ts

@@ -0,0 +1,59 @@
+import { db } from '../db/index.js';
+import { nativeInstance, runtimePasswordForServer } from '../backends/native.js';
+import { getServer } from '../fleet/servers.js';
+import { listServerScripts } from '../fleet/inventory.js';
+import { recipeStorage, recipeSummaryForDeployment, type LaunchRecipePlan } from '../fleet/launch-recipes.js';
+
+/** Preserve the recorded deployed inventory, including changes made after launch. */
+export function captureMoveRecipe(server: string): LaunchRecipePlan {
+    const instance = nativeInstance(server);
+    const row = getServer(server);
+    if (!instance || !row?.map_slug) throw new Error('A live native instance and saved map are required before moving.');
+    if (!runtimePasswordForServer(server)) throw new Error('The deployed runtime credentials are unavailable.');
+    const launch = db().prepare(`SELECT d.id FROM server_deployments d
+        JOIN native_provisioning n ON n.deployment_id=d.id
+        JOIN server_launch_recipes r ON r.deployment_id=d.id
+        WHERE n.instance_id=? AND d.state IN ('done','warning') AND d.server_deleted_at IS NULL
+        ORDER BY d.rowid DESC LIMIT 1`).get(instance.id) as { id: string } | undefined;
+    const previous = launch && recipeSummaryForDeployment(launch.id)?.recipe;
+    if (!previous || previous.game !== row.game || previous.mapSlug !== row.map_slug) {
+        throw new Error('The deployed launch recipe is unavailable or differs from the saved map.');
+    }
+    const inventory = listServerScripts(server);
+    if (!inventory.length) throw new Error('The deployed file inventory is unavailable.');
+    const files = inventory.map(file => {
+        const before = previous.files.find(item => item.path === file.rel_path);
+        const version = file.script_version_id === null ? undefined : db().prepare(`SELECT v.version,v.kind,s.name
+            FROM script_versions v JOIN scripts s ON s.id=v.script_id WHERE v.id=?`).get(file.script_version_id) as
+            { version: number; kind: string; name: string } | undefined;
+        const kind = version?.kind === 'baseline' ? 'config' : version?.kind ?? before?.kind;
+        if (!['script', 'plugin', 'config'].includes(kind ?? '')) throw new Error(`The deployed file type is unavailable for ${file.rel_path}.`);
+        if (!db().prepare('SELECT 1 FROM file_blobs WHERE sha256=? AND size=?').get(file.sha256, file.size)) {
+            throw new Error(`The deployed bytes are unavailable for ${file.rel_path}.`);
+        }
+        return { name: version?.name ?? before?.name ?? file.rel_path, kind: kind as 'script' | 'plugin' | 'config',
+            path: file.rel_path, versionId: file.script_version_id, version: version?.version ?? null,
+            sha256: file.sha256, size: file.size };
+    });
+    if (!files.some(file => file.path === 'electric_cherry_map.cfg' && file.kind === 'config')) {
+        throw new Error('The deployed map configuration is unavailable.');
+    }
+    return { format: 1, game: previous.game, mapSlug: previous.mapSlug, mapName: previous.mapName,
+        expectedMap: previous.expectedMap, configFile: previous.configFile,
+        configSource: 'Preserved deployed files', capturedAt: new Date().toISOString(), image: instance.image,
+        native: { libraryId: instance.library_id, image: instance.image }, files };
+}
+
+/** Immutable hash references keep deployed bytes alive without decoding plugins on the controller. */
+export function moveRecipeStorage(recipe: LaunchRecipePlan): ReturnType<typeof recipeStorage> {
+    const metadata = { ...recipe, files: recipe.files.map(({ content: _content, ...file }) => file) };
+    const stored = recipeStorage(metadata);
+    return { recipeSummary: stored.recipeSummary,
+        recipe: JSON.stringify({ ...metadata, files: metadata.files.map(file => ({ ...file, contentHash: file.sha256 })) }) };
+}
+
+export function moveRecipeMatches(server: string, recipe: LaunchRecipePlan): boolean {
+    const inventory = listServerScripts(server);
+    return inventory.length === recipe.files.length && inventory.every(file => recipe.files.some(saved =>
+        saved.path === file.rel_path && saved.sha256 === file.sha256 && saved.size === file.size));
+}

src/lib/server/provisioning/native.ts

@@ -2,7 +2,7 @@ import { assertScalingOwnership } from '../autoscaling/ownership.js';
 import { removalAuthorization } from '../autoscaling/authorization.js';
 import { scalingIntent } from '../autoscaling/admission.js';
 import { putFileBlob, getFileBlob, blobSize } from '../db/file-blobs.js';
-import { concurrencyBlocker, preparingJobs } from './concurrency.js';
+import { concurrencyBlocker, preparingJobs, committedRebalanceStartup } from './concurrency.js';
 import { isIP } from 'node:net';
 import { setImmediate as yieldToController } from 'node:timers/promises';
 import { choosePlacement, availableHostPort, type PlacementOptions } from '../hosts/capacity.js';
@@ -15,7 +15,7 @@ import { config, relDestForKind, serverDir } from '../config.js';
 import { nativeInstance, nativeRequest, unplacedNativeCreation, type NativeInstance } from '../backends/native.js';
 import { getHost, listHosts, getHostOperationStatus, queueHostOperation, queueHostOperationSerialized, releaseHostPorts, reserveHostPort } from '../hosts/index.js';
 import { newestReadyLibraryId, assertReadyLibrary } from '../game-files/index.js';
-import { blueprintRecipe, captureRecipePlan, recipeForDeployment, recipeStorage, summarizeRecipe, type LaunchRecipePlan } from '../fleet/launch-recipes.js';
+import { blueprintRecipe, captureRecipePlan, recipeForDeployment, recipeSummaryForDeployment, recipeStorage, summarizeRecipe, type LaunchRecipePlan } from '../fleet/launch-recipes.js';
 import { prepareNativeLaunch, evaluateNativePlacement, type PreparedNativeLaunch } from './launch-preparation.js';
 import { getServerKey, reserveServerKey, releaseServerKey, availableServerKeyCount } from '../fleet/server-keys.js';
 import { activeBetaRemovalBlocker, getServer } from '../fleet/servers.js';
@@ -36,7 +36,8 @@ import type { LaunchPreview } from '../../deployment-plan.js';
 import type { ProvisioningAction } from '../../provisioning.js';
 import { hostAgentUpdatesEnabled } from '../../hosts.js';
 import { cancelCreation, finishCreationCancellation } from './cancel-creation.js';
-import { hostMigration, processMigrationRemoval, reconcileHostDrains } from './host-drain.js';
+import { reconcileHostDrains } from './host-drain.js';
+import { hostMigration, isRebalanceMigration, processMigrationRemoval } from './migrations.js';
 import { safetyOverride, safetyOverrideCapability } from './safety-override.js';
 
 export type NativeSelection = import('../../hosts.js').NativePlacement;
@@ -390,7 +391,12 @@ export async function processNativeProvisioning(eligibility: (operation: { id: s
 const activeJobs = new Map<string, Promise<void>>();
 // Preparation owns startup capacity until its command is committed or discarded.
 
-function otherPreparations(id: string): number { return preparingJobs.size - Number(preparingJobs.has(id)); }
+function otherPreparations(id: string): number {
+    if (!preparingJobs.size) return 0;
+    const reserved = new Set((db().prepare(`SELECT id FROM server_deployments WHERE id IN (SELECT value FROM json_each(?))
+        AND ${committedRebalanceStartup}`).all(JSON.stringify([...preparingJobs])) as { id: string }[]).map(row => row.id));
+    return [...preparingJobs].filter(other => other !== id && !reserved.has(other)).length;
+}
 
 let launchCommit: Promise<void> = Promise.resolve();
 function commitPreparedLaunch(commit: () => void): Promise<void> {
@@ -469,7 +475,7 @@ async function processJob(job: Job, eligibility: (job: Job) => string | null) {
         }
         const settings = fleetSettings();
         const active = db().prepare(`SELECT COUNT(*) AS n FROM server_deployments WHERE id!=? AND server_deleted_at IS NULL
-            AND action IN ('create','rebuild') AND state IN ('running','awaiting-health')`).get(job.id) as { n: number };
+            AND action IN ('create','rebuild') AND (state IN ('running','awaiting-health') OR ${committedRebalanceStartup})`).get(job.id) as { n: number };
         if (active.n + otherPreparations(job.id) >= settings.createConcurrency) { hold(job, 'Waiting for startup capacity.'); return; }
         const capacityBlocker = concurrencyBlocker(job.id, job.action);
         if (capacityBlocker) { hold(job, capacityBlocker); return; }
@@ -645,7 +651,7 @@ async function processJob(job: Job, eligibility: (job: Job) => string | null) {
     if (capacityBlocker) { hold(job, capacityBlocker); return; }
     const settings = fleetSettings();
     const count = (actions: string[]) => (db().prepare(`SELECT COUNT(*) AS n FROM server_deployments WHERE id != ? AND server_deleted_at IS NULL
-        AND action IN (${actions.map(() => '?').join(',')}) AND state IN ('running','awaiting-health')`).get(job.id, ...actions) as { n: number }).n;
+        AND action IN (${actions.map(() => '?').join(',')}) AND (state IN ('running','awaiting-health') OR ${committedRebalanceStartup})`).get(job.id, ...actions) as { n: number }).n;
     if ((job.action === 'create' || job.action === 'rebuild') && count(['create','rebuild']) + otherPreparations(job.id) >= settings.createConcurrency) {
         deploymentStage(job.id, 'Waiting for startup capacity.', { phase: 'startup-queue',
             blocker: { code: 'startup-capacity', message: 'Waiting for startup capacity.', automatic: true } }); return;
@@ -694,13 +700,15 @@ async function processJob(job: Job, eligibility: (job: Job) => string | null) {
             preparingJobs.add(job.id);
             preparedKey = getServerKey(job.server) ?? null;
             if (!preparedKey) throw new Error('The saved server key was removed.');
-            const prior = recipeForDeployment(job.id)!.recipe as NativeRecipe;
+            const frozenRecipeId = migration && isRebalanceMigration(job.id) ? job.id : undefined;
+            const prior = (frozenRecipeId ? { format: 1, ...recipeSummaryForDeployment(job.id)!.recipe }
+                : recipeForDeployment(job.id)!.recipe) as NativeRecipe;
             deploymentStage(job.id, 'Preparing launch files for the host agent', { phase: 'prepare',
                 checkpoint: { key: 'startup-queue', label: 'Startup capacity available' } });
             preparedPassword = config.rcon.password || randomBytes(32).toString('hex');
             prepared = await prepareNativeLaunch({ game: prior.game, mapSlug: prior.mapSlug, native: prior.native,
                 instance, serverKey: preparedKey, rconPassword: preparedPassword,
-                rebuild: job.action === 'rebuild' && !migration, safetyOverride: overrideSafety });
+                rebuild: job.action === 'rebuild' && !migration, safetyOverride: overrideSafety, frozenRecipeId });
             if (job.action === 'rebuild' && !migration && !await checkOccupancy()) return;
         }
         const commit = db().transaction(() => {
@@ -731,8 +739,10 @@ async function processJob(job: Job, eligibility: (job: Job) => string | null) {
             if (job.action === 'destroy' && count(['destroy']) >= fleetSettings().destroyConcurrency) { hold(job, 'Waiting for destruction capacity.'); return; }
             if (prepared) {
                 // Compare the same runtime config bytes while rechecking current promoted files.
-                if (getServerKey(job.server) !== preparedKey || !sameLaunchFiles(prepared.recipe,
-                    captureRecipePlan(prepared.recipe.game, prepared.recipe.mapSlug, preparedPassword))) {
+                const expected = migration && isRebalanceMigration(job.id)
+                    ? { format: 1 as const, ...recipeSummaryForDeployment(job.id)!.recipe }
+                    : captureRecipePlan(prepared.recipe.game, prepared.recipe.mapSlug, preparedPassword);
+                if (getServerKey(job.server) !== preparedKey || !sameLaunchFiles(prepared.recipe, expected)) {
                     hold(job, 'Launch files or credentials changed during preparation; preparing the current selection next pass.'); return;
                 }
                 assertReadyLibrary(instance!.host_id, instance!.library_id);

src/lib/server/rebalancing/admission.ts

@@ -0,0 +1,26 @@
+import { db } from '../db/index.js';
+import type { RebalanceCandidate, RebalanceSettings } from '../../rebalancing.js';
+import { rebalanceMoveBlocker } from './planner.js';
+import { rebalanceSnapshot } from './snapshot.js';
+
+/** A delivered migration must finish even after the operator stops its parent run. */
+export function rebalanceAdmission(operationId: string): string | null {
+    const operation = db().prepare('SELECT triggered_by FROM server_deployments WHERE id=?').get(operationId) as { triggered_by: string } | undefined;
+    if (!operation?.triggered_by.startsWith('rebalance:')) return null;
+    const migration = db().prepare('SELECT remove_operation_id,removed_at FROM ec_host_migrations WHERE deployment_id=?')
+        .get(operationId) as { remove_operation_id: string | null; removed_at: string | null } | undefined;
+    if (migration?.remove_operation_id || migration?.removed_at) return null;
+    const row = db().prepare(`SELECT m.candidate,r.settings,r.state FROM rebalancing_moves m
+        JOIN rebalancing_runs r ON r.id=m.run_id WHERE m.operation_id=? AND m.state='moving'`).get(operationId) as
+        { candidate: string; settings: string; state: string } | undefined;
+    if (!row || row.state !== 'running') return 'Rebalance run is no longer requesting this move.';
+    const settings = JSON.parse(row.settings) as RebalanceSettings;
+    return rebalanceMoveBlocker(rebalanceSnapshot(settings, operationId), JSON.parse(row.candidate) as RebalanceCandidate);
+}
+
+/** Cancelling an individual uncommitted move also withdraws the rest of that run. */
+export function requestRebalanceStopForOperation(operationId: string): void {
+    db().prepare(`UPDATE rebalancing_runs SET state='stopping',updated_at=?,message='Stopping after the current move.'
+        WHERE state='running' AND id IN (SELECT run_id FROM rebalancing_moves WHERE operation_id=?)`)
+        .run(new Date().toISOString(), operationId);
+}

src/lib/server/rebalancing/index.ts

@@ -0,0 +1,214 @@
+import { randomUUID } from 'node:crypto';
+import { db } from '../db/index.js';
+import { getHost } from '../hosts/index.js';
+import { requestInstanceMove } from '../provisioning/migrations.js';
+import { cancelProvisioningJob } from '../provisioning/cancellation.js';
+import { rebalanceSettings, type RebalanceCandidate, type RebalanceMove, type RebalancePreview,
+    type RebalanceReport, type RebalanceRun } from '../../rebalancing.js';
+import { planRebalance, rebalanceMoveBlocker } from './planner.js';
+import { rebalanceHosts, rebalanceSnapshot } from './snapshot.js';
+import { rebalanceAdmission } from './admission.js';
+
+export class RebalanceError extends Error { constructor(message: string, public status = 400) { super(message); } }
+interface RunRow { id: string; settings: string; state: RebalanceRun['state']; created_at: string; updated_at: string; message: string }
+interface MoveRow { id: string; candidate: string; state: RebalanceMove['state']; operation_id: string | null; message: string; updated_at: string }
+const atNow = () => new Date().toISOString();
+const movesFor = (id: string): RebalanceMove[] => (db().prepare('SELECT * FROM rebalancing_moves WHERE run_id=? ORDER BY ordinal').all(id) as MoveRow[])
+    .map(row => ({ ...JSON.parse(row.candidate), id: row.id, state: row.state, operationId: row.operation_id, message: row.message, updatedAt: row.updated_at }));
+function present(row: RunRow): RebalanceRun {
+    return { id: row.id, settings: JSON.parse(row.settings), state: row.state, createdAt: row.created_at,
+        updatedAt: row.updated_at, message: row.message, moves: movesFor(row.id) };
+}
+export function getRebalanceRun(id: string): RebalanceRun | null {
+    const row = db().prepare('SELECT * FROM rebalancing_runs WHERE id=?').get(id) as RunRow | undefined;
+    return row ? present(row) : null;
+}
+export function rebalanceReport(): RebalanceReport {
+    return { hosts: rebalanceHosts(), runs: (db().prepare(`SELECT * FROM rebalancing_runs
+        ORDER BY state IN ('running','stopping') DESC,created_at DESC,rowid DESC LIMIT 20`).all() as RunRow[]).map(present) };
+}
+
+/** Preview has no resource reservations or host side effects. */
+export function previewRebalance(input: unknown): RebalancePreview {
+    const settings = rebalanceSettings(input), now = Date.now();
+    const plan = planRebalance(rebalanceSnapshot(settings), settings);
+    const preview: RebalancePreview = { ...plan, settings, id: randomUUID(), createdAt: new Date(now).toISOString(),
+        expiresAt: new Date(now + 5 * 60_000).toISOString() };
+    db().prepare('INSERT INTO rebalancing_previews(id,preview,expires_at) VALUES(?,?,?)').run(preview.id, JSON.stringify(preview), preview.expiresAt);
+    return preview;
+}
+
+function hasOutstandingRun(): boolean {
+    return !!db().prepare(`SELECT 1 FROM rebalancing_runs WHERE state IN ('running','stopping')
+        UNION ALL SELECT 1 FROM rebalancing_moves m JOIN server_deployments d ON d.id=m.operation_id
+        WHERE d.state IN ('queued','waiting','running','awaiting-health','interrupted')
+        UNION ALL SELECT 1 FROM rebalancing_moves m JOIN native_provisioning n ON n.deployment_id=m.operation_id
+        LEFT JOIN ec_host_migrations migration ON migration.deployment_id=m.operation_id
+        JOIN ec_host_operations o ON o.id IN (n.operation_id,n.stage_operation_id,migration.remove_operation_id)
+        WHERE o.state NOT IN ('completed','failed') LIMIT 1`).get();
+}
+const moveSignature = (moves: RebalanceCandidate[]) => JSON.stringify(moves.map(move =>
+    [move.instanceId, move.generation, move.sourceHostId, move.targetHostId]));
+
+export function startRebalance(previewId: string): RebalanceRun {
+    return db().transaction(() => {
+        const row = db().prepare('SELECT preview,expires_at,run_id FROM rebalancing_previews WHERE id=?').get(previewId) as
+            { preview: string; expires_at: string; run_id: string | null } | undefined;
+        if (!row) throw new RebalanceError('Preview not found. Preview the distribution again.', 404);
+        // A retried request returns the original bounded run, even after its preview expires.
+        if (row.run_id) return getRebalanceRun(row.run_id)!;
+        if (Date.parse(row.expires_at) <= Date.now()) throw new RebalanceError('Preview expired. Preview the distribution again.', 409);
+        if (hasOutstandingRun()) throw new RebalanceError('Finish or stop the current rebalance and settle its operations before starting another.', 409);
+        const preview = JSON.parse(row.preview) as RebalancePreview;
+        const fresh = planRebalance(rebalanceSnapshot(preview.settings), preview.settings);
+        if (!fresh.moves.length || moveSignature(fresh.moves) !== moveSignature(preview.moves)) {
+            throw new RebalanceError('The fleet changed. Preview the distribution again before starting.', 409);
+        }
+        const id = randomUUID(), at = atNow();
+        db().prepare(`INSERT INTO rebalancing_runs(id,settings,state,created_at,updated_at,message)
+            VALUES(?,?,'running',?,?,'Waiting to begin the first move.')`).run(id, JSON.stringify(preview.settings), at, at);
+        for (const [ordinal, move] of fresh.moves.entries()) db().prepare(`INSERT INTO rebalancing_moves
+            (id,run_id,ordinal,server,instance_id,generation,source_host_id,target_host_id,candidate,state,message,updated_at)
+            VALUES(?,?,?,?,?,?,?,?,?,'planned','Waiting for earlier moves.',?)`)
+            .run(randomUUID(), id, ordinal, move.server, move.instanceId, move.generation, move.sourceHostId, move.targetHostId, JSON.stringify(move), at);
+        db().prepare('UPDATE rebalancing_previews SET run_id=? WHERE id=?').run(id, previewId);
+        return getRebalanceRun(id)!;
+    })();
+}
+
+function updateRun(id: string, state: RebalanceRun['state'], message: string): void {
+    db().prepare('UPDATE rebalancing_runs SET state=?,message=?,updated_at=? WHERE id=? AND (state!=? OR message!=?)')
+        .run(state, message, atNow(), id, state, message);
+}
+function updateMove(id: string, state: RebalanceMove['state'], message: string): void {
+    db().prepare('UPDATE rebalancing_moves SET state=?,message=?,updated_at=? WHERE id=? AND (state!=? OR message!=?)')
+        .run(state, message, atNow(), id, state, message);
+}
+function skipRemaining(id: string, reason: string): void {
+    db().prepare("UPDATE rebalancing_moves SET state='skipped',message=?,updated_at=? WHERE run_id=? AND state='planned'").run(reason, atNow(), id);
+}
+function committed(operationId: string): boolean {
+    return !!db().prepare('SELECT 1 FROM ec_host_migrations WHERE deployment_id=? AND (remove_operation_id IS NOT NULL OR removed_at IS NOT NULL)').get(operationId);
+}
+
+export function stopRebalance(id: string): RebalanceRun {
+    return db().transaction(() => {
+        const run = getRebalanceRun(id);
+        if (!run) throw new RebalanceError('Rebalance run not found.', 404);
+        if (!['running', 'stopping'].includes(run.state)) return run;
+        updateRun(id, 'stopping', 'Stopping after the current move.');
+        skipRemaining(id, 'Stopped before this move began.');
+        for (const move of run.moves.filter(move => move.state === 'moving')) {
+            if (move.operationId && !committed(move.operationId)
+                && cancelProvisioningJob(move.operationId, 'Rebalance stopped before source removal.')) {
+                updateMove(move.id, 'skipped', 'Stopped before source removal.');
+            }
+        }
+        if (!movesFor(id).some(move => move.state === 'moving')) updateRun(id, 'stopped', 'Stopped. Completed moves are retained.');
+        return getRebalanceRun(id)!;
+    })();
+}
+
+function refreshedAfterMove(move: RebalanceMove): boolean {
+    return [move.sourceHostId, move.targetHostId].every(id => {
+        const heartbeat = getHost(id)?.lastHeartbeat;
+        return !!heartbeat && Date.parse(heartbeat) > Date.parse(move.updatedAt);
+    });
+}
+
+/** Only a single approved move is admitted; Provisioning owns every host command. */
+function reconcileRun(run: RebalanceRun): boolean {
+    let active = run.moves.find(move => move.state === 'moving');
+    if (active) {
+        const operation = active.operationId && db().prepare(`SELECT d.state,d.stage,d.error,n.verified_at
+            FROM server_deployments d LEFT JOIN native_provisioning n ON n.deployment_id=d.id WHERE d.id=?`).get(active.operationId) as
+            { state: string; stage: string; error: string | null; verified_at: string | null } | undefined;
+        if (!operation) {
+            updateMove(active.id, 'failed', 'Provisioning operation is unavailable. Inspect the instance before starting another run.');
+            skipRemaining(run.id, 'Stopped after a move could not be reconciled.');
+            updateRun(run.id, 'failed', 'A move could not be reconciled.'); return false;
+        }
+        if (operation.state === 'cancelled') {
+            updateMove(active.id, 'skipped', operation.stage);
+            if (run.state === 'running') updateRun(run.id, 'stopping', 'An operation was cancelled; stopping the remaining moves.');
+        } else if (['done', 'warning'].includes(operation.state) && operation.verified_at) {
+            updateMove(active.id, 'done', 'Healthy on the destination host.');
+        } else if (['failed', 'warning', 'interrupted'].includes(operation.state)) {
+            updateMove(active.id, 'failed', operation.error ?? operation.stage);
+            skipRemaining(run.id, 'Stopped after a move needs attention.');
+            updateRun(run.id, 'failed', 'A move needs attention. Use its Provisioning operation to recover.'); return false;
+        } else {
+            if (active.operationId && !committed(active.operationId)) {
+                const reason = run.state === 'stopping' ? 'Rebalance stopped before source removal.' : rebalanceAdmission(active.operationId);
+                if (reason && cancelProvisioningJob(active.operationId, reason, { preserveRebalanceRun: true })) {
+                    updateMove(active.id, 'skipped', reason);
+                } else {
+                    updateMove(active.id, 'moving', operation.stage);
+                    updateRun(run.id, run.state, operation.stage); return false;
+                }
+            } else {
+                updateMove(active.id, 'moving', operation.stage);
+                updateRun(run.id, run.state, run.state === 'stopping' ? 'Finishing the current move before stopping.' : operation.stage); return false;
+            }
+        }
+        run = getRebalanceRun(run.id)!;
+        active = undefined;
+    }
+    if (run.state === 'stopping') {
+        skipRemaining(run.id, 'Stopped before this move began.');
+        updateRun(run.id, 'stopped', 'Stopped. Completed moves are retained.'); return false;
+    }
+    if (!run.moves.some(move => move.state === 'planned')) {
+        const done = run.moves.filter(move => move.state === 'done').length;
+        const skipped = run.moves.filter(move => move.state === 'skipped').length;
+        updateRun(run.id, 'done', `Finished: ${done} moved${skipped ? `, ${skipped} skipped` : ''}.`); return false;
+    }
+    const latest = run.moves.filter(move => move.state === 'done').sort((a, b) => b.updatedAt.localeCompare(a.updatedAt))[0];
+    if (latest && !refreshedAfterMove(latest)) {
+        updateRun(run.id, 'running', 'Waiting for fresh source and destination measurements after the last move.'); return false;
+    }
+    const snapshot = rebalanceSnapshot(run.settings);
+    for (const move of run.moves.filter(move => move.state === 'planned')) {
+        const reason = rebalanceMoveBlocker(snapshot, move);
+        if (reason) { updateMove(move.id, 'skipped', reason); continue; }
+        // This transaction includes the durable move-to-operation link. A restarted
+        // worker sees either both records or neither and cannot repeat admission.
+        let operationId: string;
+        try { operationId = requestInstanceMove(move.server, move.targetHostId, { triggeredBy: `rebalance:${run.id}` }); }
+        catch (error) { updateMove(move.id, 'skipped', (error as Error).message); continue; }
+        db().prepare("UPDATE rebalancing_moves SET state='moving',operation_id=?,message='Waiting for safe migration admission.',updated_at=? WHERE id=?")
+            .run(operationId, atNow(), move.id);
+        updateRun(run.id, 'running', `Moving ${move.server} to ${move.targetHostName}.`);
+        return true;
+    }
+    const done = run.moves.filter(move => move.state === 'done').length;
+    updateRun(run.id, 'done', `Finished: ${done} moved. Remaining moves no longer qualify.`);
+    return false;
+}
+
+export async function runRebalancing(): Promise<boolean> {
+    return db().transaction(() => {
+        // Recovery is explicit in Provisioning. Reflect a recovered move, but
+        // never restart the other moves from the failed parent run.
+        const recovered = db().prepare(`SELECT m.id,m.run_id FROM rebalancing_moves m
+            JOIN rebalancing_runs r ON r.id=m.run_id JOIN server_deployments d ON d.id=m.operation_id
+            JOIN native_provisioning n ON n.deployment_id=d.id
+            WHERE r.state='failed' AND m.state='failed' AND d.state IN ('done','warning') AND n.verified_at IS NOT NULL`)
+            .all() as { id: string; run_id: string }[];
+        for (const move of recovered) {
+            updateMove(move.id, 'done', 'Recovered through Provisioning; healthy on the destination host.');
+            updateRun(move.run_id, 'stopped', 'The failed move was recovered. Remaining moves stay stopped.');
+        }
+        const row = db().prepare("SELECT * FROM rebalancing_runs WHERE state IN ('running','stopping') ORDER BY created_at LIMIT 1").get() as RunRow | undefined;
+        if (!row) return false;
+        return reconcileRun(present(row));
+    })();
+}
+
+export function pruneRebalancing(): void {
+    const now = atNow(), cutoff = new Date(Date.now() - 31 * 86_400_000).toISOString();
+    db().prepare('DELETE FROM rebalancing_previews WHERE expires_at<? AND run_id IS NULL').run(now);
+    db().prepare(`DELETE FROM rebalancing_runs WHERE state IN ('done','stopped') AND updated_at<?
+        AND NOT EXISTS (SELECT 1 FROM rebalancing_moves m JOIN server_deployments d ON d.id=m.operation_id
+            WHERE m.run_id=rebalancing_runs.id AND d.state NOT IN ('done','warning','cancelled'))`).run(cutoff);
+}

src/lib/server/rebalancing/planner.ts

@@ -0,0 +1,96 @@
+import type { RebalanceCandidate, RebalancePlan, RebalanceSettings } from '../../rebalancing.js';
+
+/** Difference in the pair's peak dominant utilization, in percentage points. */
+export const REBALANCE_MIN_IMPROVEMENT = 5;
+export const REBALANCE_COOLDOWN_MS = 30 * 60_000;
+export interface Resources { cpuCores: number; memoryBytes: number }
+export interface RebalancingHost {
+    id: string; name: string; cpuCount: number; memoryTotal: number;
+    used: Resources; running: Resources; startup: Resources;
+    instances: number; ports: number; reason: string | null;
+}
+export interface RebalancingInstance {
+    server: string; instanceId: string; generation: number; sourceHostId: string;
+    game: string; relief: Resources; demand: Resources;
+    releasesPort?: boolean;
+    destinations: Record<string, string | null>;
+}
+export interface RebalancingSnapshot {
+    hosts: RebalancingHost[]; instances: RebalancingInstance[];
+    exclusions: RebalancePlan['exclusions']; warnings: string[];
+}
+
+const pressure = (host: RebalancingHost, used = host.used) => Math.max(used.cpuCores / host.cpuCount, used.memoryBytes / host.memoryTotal);
+const add = (a: Resources, b: Resources): Resources => ({ cpuCores: a.cpuCores + b.cpuCores, memoryBytes: a.memoryBytes + b.memoryBytes });
+const subtract = (a: Resources, b: Resources): Resources => ({ cpuCores: Math.max(0, a.cpuCores - b.cpuCores), memoryBytes: Math.max(0, a.memoryBytes - b.memoryBytes) });
+const maximum = (a: Resources, b: Resources): Resources => ({ cpuCores: Math.max(a.cpuCores, b.cpuCores), memoryBytes: Math.max(a.memoryBytes, b.memoryBytes) });
+
+function evaluate(source: RebalancingHost, target: RebalancingHost, instance: RebalancingInstance): { reason: string | null; improvement: number; demand: Resources } {
+    const demand = maximum(instance.demand, target.running);
+    const startup = add(target.used, maximum(demand, target.startup));
+    const improvement = 100 * (Math.max(pressure(source), pressure(target))
+        - Math.max(pressure(source, subtract(source.used, instance.relief)), pressure(target, add(target.used, demand))));
+    let reason: string | null = source.reason ?? target.reason;
+    if (!reason && source.id === target.id) reason = 'Source and destination must be different hosts.';
+    if (!reason && !(target.id in instance.destinations)) reason = 'Destination compatibility has not been verified.';
+    if (!reason) reason = instance.destinations[target.id] ?? null;
+    if (!reason && target.ports < 1) reason = 'No available UDP/TCP game port.';
+    if (!reason && startup.cpuCores > target.cpuCount * 0.95) reason = 'Destination has insufficient startup CPU capacity.';
+    if (!reason && startup.memoryBytes > target.memoryTotal - 500_000_000) reason = 'Destination has insufficient startup memory capacity.';
+    if (!reason && (!Number.isFinite(improvement) || improvement + 1e-8 < REBALANCE_MIN_IMPROVEMENT)) reason = `Move would not reduce the pair’s peak utilization by at least ${REBALANCE_MIN_IMPROVEMENT} percentage points.`;
+    return { reason, improvement, demand };
+}
+
+/** Validate the approved pair against fresh evidence without replacing it with a different move. */
+export function rebalanceMoveBlocker(snapshot: RebalancingSnapshot, move: Pick<RebalanceCandidate, 'server' | 'instanceId' | 'generation' | 'sourceHostId' | 'targetHostId'>): string | null {
+    const instance = snapshot.instances.find(item => item.server === move.server && item.instanceId === move.instanceId
+        && item.generation === move.generation && item.sourceHostId === move.sourceHostId);
+    if (!instance) return snapshot.exclusions.find(item => item.server === move.server)?.reason ?? 'The instance or its generation has changed.';
+    const source = snapshot.hosts.find(host => host.id === move.sourceHostId), target = snapshot.hosts.find(host => host.id === move.targetHostId);
+    if (!source || !target) return 'The selected source or destination is unavailable.';
+    return evaluate(source, target, instance).reason;
+}
+
+/** Greedy largest improvements minimize churn; simulated relief never changes live capacity. */
+export function planRebalance(snapshot: RebalancingSnapshot, settings: RebalanceSettings): RebalancePlan {
+    const hosts = snapshot.hosts.filter(host => settings.hostIds.includes(host.id)).map(host => ({ ...host, used: { ...host.used } }));
+    const before = new Map(hosts.map(host => [host.id, { ...host, used: { ...host.used } }]));
+    const instances = snapshot.instances.filter(instance => settings.hostIds.includes(instance.sourceHostId)
+        && (settings.game === 'all' || settings.game === instance.game));
+    const moved = new Set<string>(), moves: RebalanceCandidate[] = [];
+    const limit = Math.max(1, Math.min(20, Math.floor(settings.maxMoves) || 1));
+    while (moves.length < limit) {
+        let best: { instance: RebalancingInstance; source: RebalancingHost; target: RebalancingHost; improvement: number; demand: Resources } | undefined;
+        for (const instance of [...instances].sort((a, b) => a.server.localeCompare(b.server))) {
+            if (moved.has(instance.instanceId)) continue;
+            const source = hosts.find(host => host.id === instance.sourceHostId);
+            if (!source) continue;
+            for (const target of [...hosts].sort((a, b) => a.id.localeCompare(b.id))) {
+                const result = evaluate(source, target, instance);
+                if (!result.reason && (!best || result.improvement > best.improvement + 1e-8)) best = { instance, source, target, ...result };
+            }
+        }
+        if (!best) break;
+        const { instance, source, target, improvement, demand } = best;
+        moves.push({ server: instance.server, instanceId: instance.instanceId, generation: instance.generation,
+            sourceHostId: source.id, sourceHostName: source.name, targetHostId: target.id, targetHostName: target.name, improvement });
+        source.used = subtract(source.used, instance.relief); source.instances--;
+        if (instance.releasesPort) source.ports++;
+        target.used = add(target.used, demand); target.instances++; target.ports--;
+        moved.add(instance.instanceId);
+    }
+    const exclusions = [...snapshot.exclusions];
+    for (const instance of instances.filter(item => !moved.has(item.instanceId))) {
+        const source = hosts.find(host => host.id === instance.sourceHostId)!;
+        const results = hosts.filter(host => host.id !== source.id).map(target => ({ target, ...evaluate(source, target, instance) }));
+        const fits = results.some(result => !result.reason);
+        const reason = fits && moves.length === limit ? 'Move limit reached.' : results.length
+            ? results.map(result => `${result.target.name}: ${result.reason ?? 'A different move provided more improvement.'}`).join(' ')
+            : 'No other participating host is available.';
+        exclusions.push({ server: instance.server, reason });
+    }
+    const projection = (host: RebalancingHost) => ({ cpuPercent: host.cpuCount > 0 ? host.used.cpuCores / host.cpuCount * 100 : 0,
+        memoryPercent: host.memoryTotal > 0 ? host.used.memoryBytes / host.memoryTotal * 100 : 0, instances: host.instances });
+    return { hosts: hosts.map(host => ({ id: host.id, name: host.name, before: projection(before.get(host.id)!), after: projection(host) })),
+        moves, exclusions, warnings: [...snapshot.warnings] };
+}

src/lib/server/rebalancing/schema.ts

@@ -0,0 +1,29 @@
+import type Database from 'better-sqlite3';
+
+export function ensureRebalancingSchema(handle: Database.Database): void {
+    handle.exec(`
+        CREATE TABLE IF NOT EXISTS rebalancing_runs (
+            id TEXT PRIMARY KEY, settings TEXT NOT NULL,
+            state TEXT NOT NULL CHECK(state IN ('running','stopping','done','stopped','failed')),
+            created_at TEXT NOT NULL, updated_at TEXT NOT NULL, message TEXT NOT NULL
+        );
+        CREATE UNIQUE INDEX IF NOT EXISTS rebalancing_one_run ON rebalancing_runs((1))
+            WHERE state IN ('running','stopping');
+        CREATE TABLE IF NOT EXISTS rebalancing_moves (
+            id TEXT PRIMARY KEY, run_id TEXT NOT NULL REFERENCES rebalancing_runs(id) ON DELETE CASCADE,
+            ordinal INTEGER NOT NULL, server TEXT NOT NULL, instance_id TEXT NOT NULL, generation INTEGER NOT NULL,
+            source_host_id TEXT NOT NULL, target_host_id TEXT NOT NULL,
+            candidate TEXT NOT NULL, state TEXT NOT NULL CHECK(state IN ('planned','moving','done','skipped','failed')),
+            operation_id TEXT UNIQUE REFERENCES server_deployments(id) ON DELETE SET NULL,
+            message TEXT NOT NULL, updated_at TEXT NOT NULL, UNIQUE(run_id,ordinal)
+        );
+        CREATE INDEX IF NOT EXISTS rebalancing_move_run ON rebalancing_moves(run_id,state,ordinal);
+        CREATE INDEX IF NOT EXISTS rebalancing_move_instance ON rebalancing_moves(instance_id,state,updated_at);
+        CREATE TABLE IF NOT EXISTS rebalancing_previews (
+            id TEXT PRIMARY KEY, preview TEXT NOT NULL, expires_at TEXT NOT NULL,
+            run_id TEXT REFERENCES rebalancing_runs(id) ON DELETE CASCADE
+        );
+        CREATE INDEX IF NOT EXISTS rebalancing_preview_expiry ON rebalancing_previews(expires_at);
+        CREATE INDEX IF NOT EXISTS rebalancing_preview_run ON rebalancing_previews(run_id);
+    `);
+}

src/lib/server/rebalancing/snapshot.ts

@@ -0,0 +1,144 @@
+import { db } from '../db/index.js';
+import { listHosts, HEARTBEAT_MAX_AGE_MS, requireReadyHost } from '../hosts/index.js';
+import { capacityCalibration } from '../hosts/calibration.js';
+import { assertReadyLibrary } from '../game-files/index.js';
+import { listServers } from '../fleet/servers.js';
+import { getServerKey } from '../fleet/server-keys.js';
+import { observationReader, serverObservation } from '../fleet/status.js';
+import { latestGameSubmission, telemetryReport } from '../fleet/status-evidence.js';
+import { mapIdentity, protectedWork } from '../autoscaling/snapshot.js';
+import { captureMoveRecipe } from '../provisioning/move-recipes.js';
+import { freshStatusTimestamp } from '../../server-freshness.js';
+import type { Host } from '../../hosts.js';
+import type { NativeInstance } from '../backends/native.js';
+import type { RebalanceHostOption, RebalanceSettings } from '../../rebalancing.js';
+import { REBALANCE_COOLDOWN_MS, type RebalancingSnapshot, type Resources } from './planner.js';
+
+const HISTORY_MS = 5 * 60_000;
+const nonnegative = (value: unknown): value is number => typeof value === 'number' && Number.isFinite(value) && value >= 0;
+
+function hostBlocker(host: Host, now: number): string | null {
+    if (host.mode !== 'active' || host.decommission) return 'An active host is required.';
+    const stamp = Date.parse(host.lastHeartbeat ?? '');
+    if (!host.connected || !Number.isFinite(stamp) || stamp > now || now - stamp > HEARTBEAT_MAX_AGE_MS) return 'Waiting for a fresh host report.';
+    if (host.readiness !== 'ready') return host.readinessReasons.join(' ') || 'The host is not ready.';
+    const { cpuCount, cpuUsagePercent, memoryTotal, memoryAvailable } = host.inventory;
+    if (!nonnegative(cpuCount) || !cpuCount || !nonnegative(cpuUsagePercent) || cpuUsagePercent > 100
+        || !nonnegative(memoryTotal) || !memoryTotal || !nonnegative(memoryAvailable) || memoryAvailable > memoryTotal) return 'Waiting for valid CPU and memory measurements.';
+    return null;
+}
+
+export function rebalanceHosts(): RebalanceHostOption[] {
+    const now = Date.now();
+    return listHosts().map(host => { const reason = hostBlocker(host, now); return { id: host.id, name: host.name, eligible: !reason, reason }; });
+}
+
+/** Database evidence only: no probes, writes, reservations, or source capacity credits. */
+export function rebalanceSnapshot(settings: RebalanceSettings, exceptOperationId = ''): RebalancingSnapshot {
+    const handle = db(), now = Date.now(), at = new Date(now).toISOString(), since = new Date(now - HISTORY_MS).toISOString();
+    const hosts = listHosts().filter(host => settings.hostIds.includes(host.id));
+    const natives = handle.prepare('SELECT * FROM native_instances WHERE retired_at IS NULL').all() as NativeInstance[];
+    const operations = handle.prepare(`SELECT id,server,stage FROM server_deployments WHERE server_deleted_at IS NULL
+        AND state IN ('queued','waiting','running','awaiting-health','interrupted') AND id!=?`).all(exceptOperationId) as { id: string; server: string; stage: string }[];
+    const ownServer = exceptOperationId ? (handle.prepare('SELECT server FROM server_deployments WHERE id=?').get(exceptOperationId) as { server: string } | undefined)?.server : undefined;
+    const observe = observationReader();
+    const snapshot: RebalancingSnapshot = { hosts: [], instances: [], exclusions: [], warnings: [] };
+    for (const id of settings.hostIds) if (!hosts.some(host => host.id === id)) snapshot.warnings.push(`Participating host ${id} is unavailable.`);
+    for (const host of hosts) {
+        const reason = hostBlocker(host, now), { running, startup } = capacityCalibration(host.id);
+        if (reason) snapshot.warnings.push(`${host.name}: ${reason}`);
+        const inventory = host.inventory;
+        const valid = !reason || nonnegative(inventory.cpuCount) && inventory.cpuCount > 0 && nonnegative(inventory.cpuUsagePercent)
+            && nonnegative(inventory.memoryTotal) && inventory.memoryTotal > 0 && nonnegative(inventory.memoryAvailable) && inventory.memoryAvailable <= inventory.memoryTotal;
+        const history = handle.prepare(`SELECT AVG(cpu_usage_percent) AS cpu,AVG(memory_available) AS memory FROM ec_host_resource_samples
+            WHERE host_id=? AND sampled_at>=? AND sampled_at<=? AND cpu_count=? AND memory_total=?`)
+            .get(host.id, since, at, inventory.cpuCount ?? 0, inventory.memoryTotal ?? 0) as { cpu: number | null; memory: number | null };
+        const used: Resources = { cpuCores: valid ? inventory.cpuCount! * Math.max(inventory.cpuUsagePercent!, history.cpu ?? 0) / 100 : 0,
+            memoryBytes: valid ? inventory.memoryTotal! - Math.min(inventory.memoryAvailable!, history.memory ?? inventory.memoryAvailable!) : 0 };
+        const hostInstances = natives.filter(instance => instance.host_id === host.id), measured = new Map(inventory.instanceResources?.map(sample => [sample.instanceId, sample]));
+        const reserved = new Set<string>();
+        const reserve = (id: string, allowance: Resources) => {
+            if (reserved.has(id)) return;
+            reserved.add(id); used.cpuCores += allowance.cpuCores; used.memoryBytes += allowance.memoryBytes;
+        };
+        for (const instance of hostInstances) {
+            const sample = measured.get(instance.id);
+            if (sample?.generation === instance.generation) continue; // Already in the host measurement.
+            const latest = handle.prepare(`SELECT d.state,d.submitted_at,n.operation_id,n.stage_operation_id FROM native_provisioning n
+                JOIN server_deployments d ON d.id=n.deployment_id WHERE n.instance_id=? AND n.generation=? AND d.id!=?
+                AND d.action IN ('create','rebuild') AND d.server_deleted_at IS NULL ORDER BY d.rowid DESC LIMIT 1`)
+                .get(instance.id, instance.generation, exceptOperationId) as { state: string; submitted_at: string | null; operation_id: string | null; stage_operation_id: string | null } | undefined;
+            const accounted = handle.prepare(`SELECT 1 FROM native_provisioning n JOIN server_deployments d ON d.id=n.deployment_id
+                WHERE n.instance_id=? AND n.generation=? AND d.action IN ('create','rebuild') AND d.state IN ('done','warning') AND d.finished_at<?`)
+                .get(instance.id, instance.generation, host.lastHeartbeat ?? '');
+            if (accounted) continue;
+            reserve(instance.id, latest?.state === 'queued' && !latest.submitted_at && !latest.operation_id && !latest.stage_operation_id ? running : startup);
+        }
+        const incoming = handle.prepare(`SELECT m.instance_id,m.removed_at,i.generation FROM ec_host_migrations m
+            JOIN server_deployments d ON d.id=m.deployment_id JOIN native_instances i ON i.id=m.instance_id
+            WHERE m.target_host_id=? AND m.deployment_id!=? AND d.server_deleted_at IS NULL AND i.retired_at IS NULL
+            AND d.state NOT IN ('done','warning','cancelled')`).all(host.id, exceptOperationId) as { instance_id: string; removed_at: string | null; generation: number }[];
+        for (const migration of incoming) {
+            const generation = migration.generation + (migration.removed_at ? 0 : 1);
+            if (measured.get(migration.instance_id)?.generation !== generation) reserve(migration.instance_id, startup);
+        }
+        const ports = handle.prepare('SELECT port,bind_address FROM ec_host_ports WHERE host_id=?').all(host.id) as { port: number; bind_address: string }[];
+        const usedPorts = new Set(ports.filter(port => port.port >= host.portMin && port.port <= host.portMax
+            && (port.bind_address === host.bindAddress || ['0.0.0.0', '::'].includes(port.bind_address) || ['0.0.0.0', '::'].includes(host.bindAddress))).map(port => port.port));
+        snapshot.hosts.push({ id: host.id, name: host.name, cpuCount: valid ? inventory.cpuCount! : 0, memoryTotal: valid ? inventory.memoryTotal! : 0,
+            used, running, startup, reason, instances: hostInstances.length + incoming.filter(row => !hostInstances.some(instance => instance.id === row.instance_id)).length,
+            ports: Math.max(0, host.portMax - host.portMin + 1 - usedPorts.size) });
+    }
+    const destinations = new Map<string, Record<string, string | null>>();
+    for (const row of listServers()) {
+        const identity = mapIdentity(row);
+        if (!row.host_id || !settings.hostIds.includes(row.host_id) || settings.game !== 'all' && identity?.game !== settings.game) continue;
+        const source = hosts.find(host => host.id === row.host_id), native = natives.find(instance => instance.id === row.native_instance_id);
+        // The admitted rebuild itself must not hide the pre-removal empty evidence.
+        const observation = row.name === ownServer ? serverObservation(row, { provisioning: false, settling: false,
+            telemetry: telemetryReport(row.name), submission: latestGameSubmission(row.name) }) : observe(row);
+        let reason = protectedWork(row.name, exceptOperationId);
+        if (!reason && (!identity || !native || !row.map_slug)) reason = 'A native instance with a known map is required.';
+        if (!reason && (!source || hostBlocker(source, now))) reason = source ? hostBlocker(source, now) : 'Source host is unavailable.';
+        const operation = operations.find(item => item.server === row.name);
+        if (!reason && operation) reason = `Provisioning ${operation.id}: ${operation.stage}`;
+        if (!reason && native && handle.prepare(`SELECT 1 FROM ec_host_operations WHERE instance_id=? AND state IN ('pending','accepted','running','unknown')
+            AND kind IN ('instance.create','instance.rebuild','instance.start','instance.stop','instance.restart','instance.remove','instance.cancel-create')`).get(native.id)) reason = 'An instance lifecycle command is unsettled.';
+        if (!reason && (row.desired_state === 'stopped' || ['stopped','exited','created','dead','missing'].includes(row.container_state ?? ''))) reason = 'Stopped instances are not eligible.';
+        if (!reason && (row.last_players ?? 0) > 0 && freshStatusTimestamp(row.last_seen_at, now)
+            && (!observation.telemetry_after || Date.parse(row.last_seen_at!) > Date.parse(observation.telemetry_after))) reason = 'Players are present according to a current RCON report.';
+        if (!reason && (observation.health !== 'online' || observation.last_players === null || !freshStatusTimestamp(observation.last_seen_at, now))) reason = 'Waiting for healthy, fresh occupancy evidence.';
+        if (!reason && observation.last_players !== 0) reason = 'Players are present.';
+        if (!reason && !getServerKey(row.name)) reason = 'Save a server key before moving this instance.';
+        if (!reason && handle.prepare(`SELECT 1 FROM rebalancing_moves WHERE instance_id=? AND state='done' AND updated_at>=?`)
+            .get(native!.id, new Date(now - REBALANCE_COOLDOWN_MS).toISOString())) reason = 'This instance was rebalanced less than 30 minutes ago.';
+        const sample = source?.inventory.instanceResources?.find(item => item.instanceId === native?.id && item.generation === native.generation);
+        if (!reason && (!sample || source?.inventory.instanceResourcesError || !nonnegative(sample.cpuCores) || !nonnegative(sample.memoryBytes) || !sample.memoryBytes)) reason = 'Waiting for resource measurements for the current instance generation.';
+        if (!reason && sample!.uptimeSeconds < 300) reason = 'Waiting for this instance to finish its first five minutes of startup.';
+        if (!reason) { try { captureMoveRecipe(row.name); } catch (error) { reason = (error as Error).message; } }
+        if (reason) { snapshot.exclusions.push({ server: row.name, reason }); continue; }
+        const key = JSON.stringify([native!.library_id, native!.image]);
+        if (!destinations.has(key)) {
+            const eligibility: Record<string, string | null> = {};
+            for (const host of hosts) {
+                try { requireReadyHost(host.id, { image: native!.image, requiredBytes: 1024 ** 3 }); assertReadyLibrary(host.id, native!.library_id); eligibility[host.id] = null; }
+                catch (error) { eligibility[host.id] = (error as Error).message; }
+            }
+            destinations.set(key, eligibility);
+        }
+        const history = handle.prepare(`SELECT AVG(cpu_cores) AS cpu,AVG(memory_bytes) AS memory FROM ec_instance_resource_samples
+            WHERE host_id=? AND instance_id=? AND generation=? AND phase='running' AND bucket>=? AND bucket<=?`)
+            .get(source!.id, native!.id, native!.generation, since, at) as { cpu: number | null; memory: number | null };
+        const bindConflicts = (bind: string) => bind === source!.bindAddress || ['0.0.0.0', '::'].includes(bind) || ['0.0.0.0', '::'].includes(source!.bindAddress);
+        const portOwners = handle.prepare('SELECT instance_id,bind_address FROM ec_host_ports WHERE host_id=? AND port=?')
+            .all(source!.id, native!.port) as { instance_id: string; bind_address: string }[];
+        snapshot.instances.push({ server: row.name, instanceId: native!.id, generation: native!.generation, sourceHostId: source!.id, game: identity!.game,
+            releasesPort: native!.port >= source!.portMin && native!.port <= source!.portMax && bindConflicts(native!.bind_address)
+                && portOwners.some(owner => owner.instance_id === native!.id && bindConflicts(owner.bind_address))
+                && !portOwners.some(owner => owner.instance_id !== native!.id && bindConflicts(owner.bind_address)),
+            relief: { cpuCores: Math.min(sample!.cpuCores, history.cpu ?? sample!.cpuCores), memoryBytes: Math.min(sample!.memoryBytes, history.memory ?? sample!.memoryBytes) },
+            demand: { cpuCores: Math.max(sample!.cpuCores, history.cpu ?? sample!.cpuCores), memoryBytes: Math.max(sample!.memoryBytes, history.memory ?? sample!.memoryBytes) },
+            destinations: destinations.get(key)! });
+    }
+    return snapshot;
+}

src/lib/server/retention.ts

@@ -56,6 +56,7 @@ function policies(): Policy[] {
             AND NOT EXISTS (SELECT 1 FROM watchdog_actions s WHERE s.operation_id=o.id)
             AND NOT EXISTS (SELECT 1 FROM native_creation_cancellations c WHERE o.id IN (c.creation_id,c.cleanup_id))
             AND NOT EXISTS (SELECT 1 FROM server_creation_dependencies c WHERE c.removal_id=o.id)
+            AND NOT EXISTS (SELECT 1 FROM rebalancing_moves m WHERE m.operation_id=o.id AND m.state IN ('moving','failed'))
             AND NOT EXISTS (SELECT 1 FROM server_collection_members m JOIN server_collection_members sibling ON sibling.collection_id=m.collection_id
                 JOIN server_deployments d ON d.id=sibling.deployment_id WHERE m.deployment_id=o.id AND (${unsettledDeployment}))`,
             bytes: "COALESCE(length(o.observation_json),0)+(SELECT COALESCE(SUM(length(recipe)),0) FROM server_launch_recipes WHERE deployment_id=o.id)" },
@@ -88,6 +89,7 @@ function policies(): Policy[] {
         { category: 'workflow-payload', label: 'Completed launch file bodies', table: 'server_deployments', id: 'id', clock: 'finished_at', days: 'payloadKeepDays',
             failed: "o.state='failed'",
             eligible: `o.state IN ${terminal} AND NOT (${unsettledDeployment.replaceAll('d.', 'o.')})
+                AND NOT EXISTS (SELECT 1 FROM rebalancing_moves m WHERE m.operation_id=o.id AND m.state IN ('moving','failed'))
                 AND NOT EXISTS (SELECT 1 FROM retention_controls c WHERE c.category='provisioning' AND c.record_id=o.id AND c.protected=1)
                 AND NOT EXISTS (SELECT 1 FROM retention_expirations e WHERE e.category='workflow-payload' AND e.record_id=o.id)`,
             bytes: "(SELECT COALESCE(SUM(b.size),0) FROM file_blob_references r JOIN file_blobs b ON b.sha256=r.sha256 WHERE r.owner_type='recipe' AND r.owner_id=o.id)" },

src/lib/ui/components/capacity/RebalancePanel.svelte

@@ -0,0 +1,160 @@
+<script lang="ts">
+    import { onMount, untrack } from 'svelte';
+    import type { RebalancePreview, RebalanceReport, RebalanceRun, RebalanceSettings } from '$lib/rebalancing.js';
+    import { pollVisible } from '$ui/poll.js';
+    import { timeFormat } from '$ui/stores/time-zone.js';
+    import { rebalanceRunLabels } from '$ui/rebalancing-presentation.js';
+    import Button from '$ui/components/primitives/Button.svelte';
+    import Freshness from '$ui/components/Freshness.svelte';
+    import RebalancePreviewView from './RebalancePreviewView.svelte';
+    import RebalanceRunView from './RebalanceRunView.svelte';
+
+    let { canManage, preferredHost = null, preferredRun = null }: {
+        canManage: boolean; preferredHost?: string | null; preferredRun?: string | null;
+    } = $props();
+    let report = $state<RebalanceReport | null>(null);
+    let selectedHosts = $state<string[]>([]);
+    let game = $state<RebalanceSettings['game']>('all');
+    let maxMoves = $state<number | undefined>(5);
+    let preview = $state<RebalancePreview | null>(null);
+    let selectedRun = $state(untrack(() => preferredRun));
+    let busy = $state<'preview' | 'start' | 'stop' | null>(null);
+    let loading = $state(false);
+    let refreshError = $state('');
+    let actionError = $state('');
+    let notice = $state('');
+    let now = $state(untrack(() => Date.now()));
+    let checkedAt = $state<number | null>(null);
+    let initialized = false;
+    let sequence = 0;
+    let loadRequest: AbortController | null = null;
+    let actionRequest: AbortController | null = null;
+    const activeRun = $derived(report?.runs.find(run => run.state === 'running' || run.state === 'stopping') ?? null);
+    const run = $derived(report?.runs.find(item => item.id === selectedRun) ?? activeRun ?? report?.runs[0] ?? null);
+    const validSettings = $derived(selectedHosts.length >= 2 && selectedHosts.length <= 100 && Number.isInteger(maxMoves) && maxMoves! >= 1 && maxMoves! <= 20);
+    const expired = $derived(!!preview && Date.parse(preview.expiresAt) <= now);
+    function editSettings() { preview = null; actionError = ''; notice = ''; }
+    function toggleHost(id: string, checked: boolean) {
+        selectedHosts = checked ? [...selectedHosts, id] : selectedHosts.filter(value => value !== id);
+        editSettings();
+    }
+    function cancelLoad() { sequence++; loadRequest?.abort(); loadRequest = null; loading = false; }
+    async function load(signal?: AbortSignal) {
+        if (busy || loading) return;
+        const current = ++sequence;
+        const controller = new AbortController();
+        loadRequest = controller; loading = true;
+        try {
+            const response = await fetch('/api/rebalancing', { signal: AbortSignal.any([controller.signal, AbortSignal.timeout(20_000), ...(signal ? [signal] : [])]) });
+            const body = await response.json();
+            if (!response.ok) throw new Error(body.error ?? 'Could not refresh rebalancing.');
+            if (current !== sequence) return;
+            report = body as RebalanceReport; refreshError = ''; checkedAt = Date.now(); now = Date.now();
+            if (!initialized) {
+                selectedHosts = preferredHost ? report.hosts.filter(host => host.id === preferredHost && host.eligible).map(host => host.id)
+                    : report.hosts.filter(host => host.eligible).map(host => host.id).slice(0, 100);
+                initialized = true;
+            } else {
+                const eligible = new Set(report.hosts.filter(host => host.eligible).map(host => host.id));
+                const available = selectedHosts.filter(id => eligible.has(id));
+                if (available.length !== selectedHosts.length) {
+                    selectedHosts = available; preview = null;
+                    notice = 'Host availability changed. Unavailable hosts were removed from your selection; preview again before starting.';
+                }
+            }
+        } catch (cause) {
+            if (current === sequence && !controller.signal.aborted && !signal?.aborted) refreshError = (cause as Error).message;
+            return false;
+        } finally { if (current === sequence) { loadRequest = null; loading = false; } }
+    }
+    async function act(action: 'preview' | 'start' | 'stop') {
+        if (busy || !canManage) return;
+        const payload = action === 'preview' ? { action, settings: { hostIds: selectedHosts, game, maxMoves } }
+            : action === 'start' ? { action, previewId: preview?.id } : { action, id: run?.id };
+        cancelLoad(); busy = action; actionError = ''; notice = '';
+        if (action === 'preview') preview = null;
+        const controller = new AbortController(); actionRequest = controller;
+        try {
+            const response = await fetch('/api/rebalancing', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(payload), signal: AbortSignal.any([controller.signal, AbortSignal.timeout(30_000)]) });
+            const body = await response.json();
+            if (!response.ok) throw new Error(body.error ?? 'Rebalance request failed.');
+            if (action === 'preview') preview = body.preview;
+            else {
+                const updated = body.run as RebalanceRun;
+                if (report) report = { ...report, runs: [updated, ...report.runs.filter(item => item.id !== updated.id)] };
+                selectedRun = updated.id; preview = null;
+                notice = action === 'start' ? 'Rebalance started. Moves run one at a time.' : 'Unstarted moves have been withdrawn. Any move already in progress will finish.';
+            }
+            now = Date.now();
+        } catch (cause) {
+            if (!controller.signal.aborted) actionError = `${(cause as Error).message}${action === 'preview' ? '' : ' Refresh the run status before retrying an unconfirmed request.'}`;
+        } finally {
+            if (!controller.signal.aborted) { busy = null; actionRequest = null; await load(); }
+        }
+    }
+    onMount(() => {
+        const stop = pollVisible(load, 10_000);
+        const timer = setInterval(() => now = Date.now(), 5_000);
+        return () => { stop(); clearInterval(timer); cancelLoad(); actionRequest?.abort(); };
+    });
+</script>
+
+<section class="panel" aria-labelledby="rebalance-title">
+    <div class="heading"><div><h2 id="rebalance-title">Rebalance instances</h2><p>Spread existing instances across hosts according to CPU and memory use relative to each host’s capacity.</p></div>
+        {#if checkedAt && !refreshError}<Freshness updatedAt={checkedAt} intervalLabel="Updates every 10 seconds" />{/if}
+    </div>
+    <p class="help">Only healthy, empty lobbies can move. Each move rebuilds the lobby on another host, causing a temporary outage; its address or port may change. The server identity, map, key, and recorded deployed files are retained. Local mutable files are not copied.</p>
+    {#if refreshError}<div class="error" role="alert"><p>{refreshError}{report ? ' Showing the last loaded rebalancing status.' : ''}</p><Button size="sm" disabled={loading || !!busy} onclick={() => void load()}>Retry</Button></div>{/if}
+    {#if actionError}<p class="error" role="alert">{actionError}</p>{/if}
+    {#if notice}<p class="notice" role="status">{notice}</p>{/if}
+    {#if !report && loading}<p role="status">Loading hosts and rebalance runs…</p>{/if}
+    {#if report}
+        {#if canManage}
+            <form onsubmit={event => { event.preventDefault(); void act('preview'); }}>
+                <fieldset disabled={!!busy || !!activeRun}>
+                    <legend>Participating hosts</legend>
+                    <p class="help">Choose at least two hosts. Existing lobbies may move between any selected hosts.</p>
+                    {#if preferredHost && !report.hosts.some(host => host.id === preferredHost && host.eligible)}<p class="help">The host opened from the shortcut is currently unavailable. Choose eligible hosts below.</p>{/if}
+                    <div class="host-selection">{#each report.hosts as host (host.id)}
+                        <label class:unavailable={!host.eligible}><input type="checkbox" checked={selectedHosts.includes(host.id)} disabled={!host.eligible} onchange={event => toggleHost(host.id, event.currentTarget.checked)} />
+                            <span><strong>{host.name}</strong>{#if host.reason}<small>{host.reason}</small>{/if}</span></label>
+                    {/each}</div>
+                    <div class="settings"><label>Game<select bind:value={game} onchange={editSettings}><option value="all">All games</option><option value="t5">Black Ops I</option><option value="t6">Black Ops II</option></select></label>
+                        <label>Maximum moves<input type="number" min="1" max="20" step="1" required bind:value={maxMoves} oninput={editSettings} /></label>
+                        <Button type="submit" disabled={!validSettings || !!busy || !!activeRun || !!refreshError} loading={busy === 'preview'}>Preview rebalance</Button>
+                    </div>
+                </fieldset>
+            </form>
+            {#if activeRun}<p class="help">A rebalance is {activeRun.state === 'stopping' ? 'finishing its current move' : 'already in progress'}. A new run can start after it finishes.{#if activeRun.id !== run?.id} <button class="text-button" onclick={() => selectedRun = activeRun.id}>View active run</button>{/if}</p>{/if}
+        {:else}<p class="help">Operators and administrators can preview, start, and stop rebalance runs.</p>{/if}
+        {#if preview}<RebalancePreviewView {preview} {expired} disabled={!!busy || !!activeRun || !!refreshError} starting={busy === 'start'} onstart={() => void act('start')} />{/if}
+        <section class="runs" aria-labelledby="rebalance-runs-title">
+            <div class="heading"><h3 id="rebalance-runs-title">Rebalance runs</h3>{#if report.runs.length}<label class="run-picker"><span class="sr-only">Select rebalance run</span><select value={run?.id ?? ''} onchange={event => selectedRun = event.currentTarget.value}>{#each report.runs as item (item.id)}<option value={item.id}>{$timeFormat.dateTime(item.createdAt)} · {rebalanceRunLabels[item.state]}</option>{/each}</select></label>{/if}</div>
+            {#if preferredRun && !report.runs.some(item => item.id === preferredRun)}<p class="help">The linked run is not in the recent history.</p>{/if}
+            {#if run}<RebalanceRunView {run} {canManage} busy={!!busy} stopping={busy === 'stop'} onstop={() => void act('stop')} />
+            {:else}<p class="help">No rebalance runs yet.</p>{/if}
+        </section>
+    {/if}
+</section>
+
+<style>
+    .panel { display:grid; gap:var(--space-5); padding:24px; border:1px solid var(--border-strong); border-radius:var(--radius-md); background:var(--surface); min-width:0; }
+    h2,h3,p { margin:0; } h2 { font-size:var(--text-lg); } h3 { font-size:var(--text-md); } p { font-size:var(--text-sm); line-height:1.6; } .heading>div>p { margin-top:8px; color:var(--text-secondary); }
+    .heading,.settings { display:flex; align-items:center; justify-content:space-between; flex-wrap:wrap; gap:12px; }
+    .help { font-size:var(--text-xs); color:var(--text-secondary); line-height:1.6; }
+    fieldset { border:0; padding:0; margin:0; display:grid; gap:12px; min-width:0; } legend { font-size:var(--text-sm); font-weight:600; padding:0 0 8px; }
+    .host-selection { display:grid; grid-template-columns:repeat(auto-fit,minmax(min(100%,220px),1fr)); gap:8px; }
+    .host-selection label { display:flex; align-items:flex-start; gap:10px; padding:12px; border:1px solid var(--border-subtle); border-radius:var(--radius-sm); cursor:pointer; }
+    .host-selection input { margin:3px 0 0; accent-color:var(--accent-primary); }
+    .host-selection span { display:grid; gap:4px; min-width:0; } .host-selection strong { font-size:var(--text-sm); font-weight:500; overflow-wrap:anywhere; }
+    .host-selection small { font-size:var(--text-xs); color:var(--text-secondary); line-height:1.5; } .host-selection .unavailable { opacity:0.65; cursor:default; }
+    .settings { justify-content:flex-start; align-items:end; } .settings label { display:grid; gap:8px; font-size:var(--text-xs); } .settings input { width:110px; }
+    select,input[type='number'] { box-sizing:border-box; padding:9px 12px; color:var(--text-primary); background:var(--surface-sunken); border:1px solid var(--border-control); border-radius:var(--radius-sm); font:var(--text-sm) var(--font-body); min-width:0; }
+    input:disabled,select:disabled { opacity:0.6; } .run-picker { max-width:100%; } .run-picker select { max-width:100%; }
+    .runs { display:grid; grid-template-columns:minmax(0,1fr); gap:16px; border-top:1px solid var(--border-subtle); padding-top:20px; min-width:0; }
+    .error { display:flex; gap:12px; align-items:center; justify-content:space-between; padding:12px 16px; color:var(--state-danger); border:1px solid var(--state-danger); border-radius:var(--radius-sm); font-size:var(--text-sm); overflow-wrap:anywhere; } .notice { color:var(--state-online); }
+    .text-button { color:var(--accent-primary-hover); } .text-button { border:0; padding:0; background:none; font:inherit; cursor:pointer; text-decoration:underline; }
+    button:focus-visible,input:focus-visible,select:focus-visible { outline:2px solid var(--accent-primary); outline-offset:3px; }
+    .sr-only { position:absolute; width:1px; height:1px; overflow:hidden; clip-path:inset(50%); white-space:nowrap; }
+    @media(max-width:600px) { .panel { padding:16px; } }
+</style>

src/lib/ui/components/capacity/RebalancePreviewView.svelte

@@ -0,0 +1,41 @@
+<script lang="ts">
+    import type { RebalancePreview } from '$lib/rebalancing.js';
+    import { contextHref } from '$lib/navigation-context.js';
+    import { timeFormat } from '$ui/stores/time-zone.js';
+    import { plural } from '$ui/plural.js';
+    import Button from '$ui/components/primitives/Button.svelte';
+    let { preview, expired, disabled = false, starting = false, onstart }: {
+        preview: RebalancePreview; expired: boolean; disabled?: boolean; starting?: boolean; onstart: () => void;
+    } = $props();
+    const numbers = new Intl.NumberFormat('en-US', { maximumFractionDigits: 1 });
+    function percent(value: number) { return `${numbers.format(value)}%`; }
+</script>
+
+<section class="preview" aria-labelledby="rebalance-preview-title">
+    <div class="heading"><h3 id="rebalance-preview-title">Preview · {preview.moves.length} proposed {plural(preview.moves.length, 'move')}</h3><span class="help">Created {$timeFormat.dateTime(preview.createdAt)}</span></div>
+    <!-- svelte-ignore a11y_no_noninteractive_tabindex (Scrollable tables must be reachable by keyboard.) -->
+    <div class="table-wrap" role="region" aria-label="Projected host distribution" tabindex="0"><table><thead><tr><th>Host</th><th>Instances</th><th>CPU</th><th>Memory</th></tr></thead><tbody>
+        {#each preview.hosts as host (host.id)}<tr><td>{host.name}</td><td>{host.before.instances} → {host.after.instances}</td><td>{percent(host.before.cpuPercent)} → {percent(host.after.cpuPercent)}</td><td>{percent(host.before.memoryPercent)} → {percent(host.after.memoryPercent)}</td></tr>{/each}
+    </tbody></table></div>
+    <p class="help">Current → projected distribution. CPU and memory estimates include the full load on each participating host.</p>
+    {#if preview.moves.length}<ol class="moves">{#each preview.moves as move}<li><a href={contextHref(`/servers/${encodeURIComponent(move.server)}`, '/capacity?rebalance=1')}>{move.server}</a><span>{move.sourceHostName} → {move.targetHostName}</span></li>{/each}</ol>
+    {:else}<p>No eligible moves would improve the distribution enough. Occupied lobbies and other excluded instances stay on their current hosts.</p>{/if}
+    {#if preview.warnings.length}<ul class="warnings">{#each preview.warnings as warning}<li>{warning}</li>{/each}</ul>{/if}
+    {#if preview.exclusions.length}<details><summary>{preview.exclusions.length} excluded {plural(preview.exclusions.length, 'instance')}</summary><ul class="exclusions">{#each preview.exclusions as exclusion}<li><strong>{exclusion.server}</strong><span>{exclusion.reason}</span></li>{/each}</ul></details>{/if}
+    <div class="start"><Button variant="primary" disabled={!preview.moves.length || disabled || expired} loading={starting} onclick={onstart}>Start rebalance</Button><p class="help">{expired ? 'This preview has expired. Preview again before starting.' : 'The controller rechecks occupancy and capacity before each move. Moves that no longer help are skipped.'}</p></div>
+</section>
+
+<style>
+    .preview { display:grid; gap:16px; border-top:1px solid var(--border-subtle); padding-top:20px; min-width:0; }
+    h3,p { margin:0; } h3 { font-size:var(--text-md); } p { font-size:var(--text-sm); line-height:1.6; }
+    .heading,.start { display:flex; align-items:center; justify-content:space-between; flex-wrap:wrap; gap:12px; }
+    .help { font-size:var(--text-xs); color:var(--text-secondary); line-height:1.6; }
+    .table-wrap { overflow-x:auto; border:1px solid var(--border-subtle); border-radius:var(--radius-sm); }
+    table { width:100%; min-width:520px; border-collapse:collapse; text-align:left; font-size:var(--text-sm); font-variant-numeric:tabular-nums; }
+    th,td { padding:12px 16px; } th { background:var(--surface-sunken); color:var(--text-secondary); font-size:var(--text-xs); font-weight:500; } td { border-top:1px solid var(--border-subtle); }
+    .moves,.exclusions { margin:0; padding-left:20px; display:grid; gap:12px; font-size:var(--text-sm); } .moves span,.exclusions span { display:block; margin-top:4px; color:var(--text-secondary); font-size:var(--text-xs); } .exclusions { margin-top:12px; } .exclusions strong { font-weight:500; }
+    .warnings { border-left:2px solid var(--state-warning); padding:0 0 0 22px; margin:0; color:var(--text-secondary); font-size:var(--text-xs); line-height:1.7; }
+    summary { cursor:pointer; color:var(--text-secondary); font-size:var(--text-sm); }
+    .start { justify-content:flex-start; } .start p { flex:1 1 280px; }
+    a { color:var(--accent-primary-hover); } a:focus-visible,summary:focus-visible,.table-wrap:focus-visible { outline:2px solid var(--accent-primary); outline-offset:3px; }
+</style>

src/lib/ui/components/capacity/RebalanceRunView.svelte

@@ -0,0 +1,27 @@
+<script lang="ts">
+    import type { RebalanceRun } from '$lib/rebalancing.js';
+    import { contextHref } from '$lib/navigation-context.js';
+    import { rebalanceMoveLabels, rebalanceRunLabels, rebalanceRunHref } from '$ui/rebalancing-presentation.js';
+    import Button from '$ui/components/primitives/Button.svelte';
+    import Badge from '$ui/components/primitives/Badge.svelte';
+    let { run, canManage, busy = false, stopping = false, onstop }: {
+        run: RebalanceRun; canManage: boolean; busy?: boolean; stopping?: boolean; onstop: () => void;
+    } = $props();
+</script>
+
+<div class="heading"><div class="run-status"><Badge variant={run.state === 'failed' ? 'danger' : run.state === 'done' ? 'online' : run.state === 'stopped' ? 'neutral' : 'info'}>{rebalanceRunLabels[run.state]}</Badge><span>{run.moves.filter(move => move.state === 'done').length} of {run.moves.length} moves completed</span><a href={rebalanceRunHref(run.id)}>Link to run</a></div>
+    {#if canManage && run.state === 'running'}<Button disabled={busy} loading={stopping} onclick={onstop}>Stop remaining moves</Button>{/if}</div>
+<p role="status">{run.message}</p>
+{#if run.state === 'running' || run.state === 'stopping'}<p class="help">Stopping withdraws unstarted moves. A move already in progress continues through health verification.</p>{/if}
+<ol class="run-moves">{#each run.moves as move (move.id)}<li><div class="heading"><div class="move-identity"><strong>{move.server}</strong><span>{move.sourceHostName} → {move.targetHostName}</span></div><Badge variant={move.state === 'failed' ? 'danger' : move.state === 'done' ? 'online' : move.state === 'moving' ? 'info' : 'neutral'}>{rebalanceMoveLabels[move.state]}</Badge></div><p>{move.message}</p>{#if move.operationId}<a href={contextHref(`/provisioning/${encodeURIComponent(move.operationId)}`, rebalanceRunHref(run.id))}>View provisioning →</a>{/if}</li>{/each}</ol>
+
+<style>
+    p { margin:0; font-size:var(--text-sm); line-height:1.6; }
+    .heading,.run-status { display:flex; align-items:center; justify-content:space-between; flex-wrap:wrap; gap:12px; }
+    .help { font-size:var(--text-xs); color:var(--text-secondary); line-height:1.6; }
+    .run-status { justify-content:flex-start; font-size:var(--text-xs); color:var(--text-secondary); } .run-moves { margin:0; padding:0; list-style:none; display:grid; gap:12px; }
+    .run-moves li { display:grid; gap:8px; padding:16px; border:1px solid var(--border-subtle); border-radius:var(--radius-sm); } .run-moves p,.run-moves a { font-size:var(--text-xs); } .run-moves p { color:var(--text-secondary); }
+    .move-identity { display:grid; gap:5px; font-size:var(--text-sm); min-width:0; overflow-wrap:anywhere; } .move-identity strong { font-weight:500; } .move-identity span { font-size:var(--text-xs); color:var(--text-secondary); }
+    a { color:var(--accent-primary-hover); } a:focus-visible { outline:2px solid var(--accent-primary); outline-offset:3px; }
+    @media(max-width:600px) { .run-moves li { padding:12px; } }
+</style>

src/lib/ui/provisioning-presentation.ts

@@ -119,6 +119,12 @@ export function provisioningReturn(value: string | null | undefined): { href: st
             }
             return { href: '/provisioning' + (params.size ? `?${params}` : ''), label: 'Back to Provisioning' };
         }
+        if (url.pathname === '/capacity' && url.searchParams.get('rebalance') === '1') {
+            const params = new URLSearchParams({ rebalance: '1' });
+            const run = url.searchParams.get('rebalanceRun');
+            if (run) params.set('rebalanceRun', run);
+            return { href: `/capacity?${params}`, label: 'Back to rebalance' };
+        }
         const match = url.pathname.match(/^\/(campaigns|betas|maintenance|collections\/launches)\/[^/]+$/);
         if (match) return { href: url.pathname, label: `Back to ${match[1] === 'collections/launches' ? 'collection' : match[1] === 'campaigns' ? 'campaign' : match[1] === 'betas' ? 'beta' : 'maintenance'}` };
     } catch { /* Fall back to the local list. */ }
@@ -180,7 +186,7 @@ export function isProvisioningDetail(value: unknown, id: string): value is Provi
     if (!record(value.activity) || !number(value.activity.total) || !(value.activity.latestId === null || number(value.activity.latestId))
         || !(value.activity.failureExcerpt === null || event(value.activity.failureExcerpt))) return false;
     if (!Array.isArray(value.sources) || !value.sources.every(source => record(source)
-        && ['campaign', 'beta', 'maintenance', 'watchdog', 'autoscaling'].includes(source.kind as string)
+        && ['campaign', 'beta', 'maintenance', 'watchdog', 'autoscaling', 'rebalance'].includes(source.kind as string)
         && text(source.source_id) && text(source.label) && localLink(source.href))) return false;
     if (!(value.collection === null || record(value.collection) && text(value.collection.id) && text(value.collection.name))) return false;
     if (!record(value.capacity) || !timestamp(value.capacity.evaluatedAt) || !Array.isArray(value.capacity.pools)

src/lib/ui/rebalancing-presentation.ts

@@ -0,0 +1,9 @@
+import type { RebalanceMove, RebalanceRun } from '$lib/rebalancing.js';
+
+export const rebalanceRunLabels: Record<RebalanceRun['state'], string> = {
+    running: 'In progress', stopping: 'Stopping', done: 'Finished', stopped: 'Stopped', failed: 'Needs attention'
+};
+export const rebalanceMoveLabels: Record<RebalanceMove['state'], string> = {
+    planned: 'Not started', moving: 'Moving', done: 'Completed', skipped: 'Skipped', failed: 'Needs attention'
+};
+export function rebalanceRunHref(id: string) { return `/capacity?rebalance=1&rebalanceRun=${encodeURIComponent(id)}`; }

src/routes/api/deployments/[id]/cancel/+server.ts

@@ -1,9 +1,10 @@
 import { json } from '@sveltejs/kit';
-import { cancelProvisioningJob } from '$lib/server/provisioning/cancellation.js';
+import { cancelProvisioningJob, rebalanceCancellationBlocker } from '$lib/server/provisioning/cancellation.js';
 import { kickProvisioning } from '$lib/server/pipeline/worker.js';
 
 export function POST({ params }: { params: { id: string } }) {
-    if (!cancelProvisioningJob(params.id)) return json({ error: 'This operation can no longer be cancelled. Refresh to see its current state.' }, { status: 409 });
+    if (!cancelProvisioningJob(params.id)) return json({ error: rebalanceCancellationBlocker(params.id)
+        ?? 'This operation can no longer be cancelled. Refresh to see its current state.' }, { status: 409 });
     void kickProvisioning();
     return json({ ok: true });
 }

src/routes/api/rebalancing/+server.ts

@@ -0,0 +1,14 @@
+import { json } from '@sveltejs/kit';
+import { RebalanceError, previewRebalance, rebalanceReport, startRebalance, stopRebalance } from '$lib/server/rebalancing/index.js';
+
+export function GET() { return json(rebalanceReport()); }
+export async function POST({ request }: { request: Request }) {
+    try {
+        const body = await request.json();
+        if (!body || typeof body !== 'object' || Array.isArray(body)) throw new RebalanceError('Provide a rebalance action.');
+        if (body.action === 'preview') return json({ preview: previewRebalance(body.settings) });
+        if (body.action === 'start' && typeof body.previewId === 'string' && body.previewId.length <= 100) return json({ run: startRebalance(body.previewId) });
+        if (body.action === 'stop' && typeof body.id === 'string' && body.id.length <= 100) return json({ run: stopRebalance(body.id) });
+        throw new RebalanceError('Choose a supported rebalance action.');
+    } catch (error) { return json({ error: (error as Error).message }, { status: error instanceof RebalanceError ? error.status : 400 }); }
+}

src/routes/capacity/+page.svelte

@@ -12,6 +12,7 @@
     import Select from '$ui/components/primitives/Select.svelte';
     import CapacityHosts from '$ui/components/capacity/CapacityHosts.svelte';
     import CapacityChart from '$ui/components/capacity/CapacityChart.svelte';
+    import RebalancePanel from '$ui/components/capacity/RebalancePanel.svelte';
     import Freshness from '$ui/components/Freshness.svelte';
     import { plural } from '$ui/plural.js';
 
@@ -21,11 +22,13 @@
     let navigating = $state(false);
     let refreshError = $state('');
     let navigationError = $state('');
+    let rebalanceOpen = $state(untrack(() => page.url.searchParams.get('rebalance') === '1'));
     let checkedAt = $state(untrack(() => Date.now()));
     let request: AbortController | null = null;
     let sequence = 0;
     const report = $derived(snapshot ?? data);
     const summary = $derived(report.current.summary);
+    const canRebalance = $derived(['admin', 'operator'].includes(page.data?.user?.role ?? ''));
     const scopeName = $derived(report.query.hostId ? report.current.hostOptions.find(host => host.id === report.query.hostId)?.name ?? 'Selected host' : 'All hosts');
     const hostOptions = $derived([{ value: '', label: 'All hosts' }, ...report.current.hostOptions.map(host => ({ value: host.id, label: host.name })),
         ...(report.query.hostId && !report.current.hostOptions.some(host => host.id === report.query.hostId) ? [{ value: report.query.hostId, label: 'Unavailable host' }] : [])]);
@@ -43,6 +46,10 @@
         const params = queryParams({ ...report.query, ...change });
         const returnTo = page.url.searchParams.get('returnTo');
         if (returnTo) params.set('returnTo', returnTo);
+        for (const key of ['rebalance', 'rebalanceHost', 'rebalanceRun']) {
+            const value = page.url.searchParams.get(key);
+            if (value) params.set(key, value);
+        }
         return `/capacity?${params}`;
     }
     function cancel() { sequence++; request?.abort(); request = null; refreshing = false; }
@@ -71,15 +78,17 @@
     beforeNavigate(() => { cancel(); navigating = true; });
     afterNavigate(() => { navigating = false; });
     $effect(() => { void data; untrack(() => { cancel(); snapshot = null; refreshError = ''; checkedAt = Date.now(); }); });
+    $effect(() => { if (page.url.searchParams.get('rebalance') === '1') rebalanceOpen = true; });
     onMount(() => { const stop = pollVisible(refresh, 30_000, false); return () => { stop(); cancel(); }; });
 </script>
 
 <svelte:head><title>Capacity · Electric Cherry</title><meta name="description" content="Available game-server capacity, resource history, and host limits." /></svelte:head>
 
 <div class="page" aria-busy={navigating}>
-    <SectionHeader title="Capacity" />
+    <SectionHeader title="Capacity">{#snippet actions()}<Button onclick={() => rebalanceOpen = !rebalanceOpen}>{rebalanceOpen ? 'Hide rebalance' : 'Rebalance'}</Button>{/snippet}</SectionHeader>
     {#if page.url.searchParams.has('returnTo')}<BackLink fallback="/hosts" />{/if}
     {#if refreshError || navigationError}<div class="refresh-error" role="alert"><strong>{navigationError || refreshError}</strong><span>Showing the last successfully loaded capacity report.</span>{#if refreshError}<Button size="sm" disabled={refreshing || navigating} onclick={() => void refresh()}>Retry</Button>{/if}</div>{/if}
+    {#if rebalanceOpen}{#key `${page.url.searchParams.get('rebalanceHost') ?? ''}:${page.url.searchParams.get('rebalanceRun') ?? ''}`}<RebalancePanel canManage={canRebalance} preferredHost={page.url.searchParams.get('rebalanceHost')} preferredRun={page.url.searchParams.get('rebalanceRun')} />{/key}{/if}
     <div class="toolbar">
         <div class="filters"><Field label="Host"><Select value={report.query.hostId ?? ''} options={hostOptions} disabled={navigating} onchange={event => void changeScope({ hostId: (event.currentTarget as HTMLSelectElement).value || null })} /></Field>
             <Field label="Game"><Select value={report.query.game} options={gameOptions} disabled={navigating} onchange={event => void changeScope({ game: (event.currentTarget as HTMLSelectElement).value as CapacityGame })} /></Field></div>

src/routes/hosts/+page.svelte

@@ -49,7 +49,7 @@
 <svelte:head><title>Hosts · Electric Cherry</title></svelte:head>
 <section class="page">
     <SectionHeader title="Hosts">
-        {#snippet actions()}{#if canManage}<Button variant="primary" onclick={() => addOpen = true}>Add host</Button>{/if}{/snippet}
+        {#snippet actions()}<Button href="/capacity?rebalance=1">Rebalance</Button>{#if canManage}<Button variant="primary" onclick={() => addOpen = true}>Add host</Button>{/if}{/snippet}
     </SectionHeader>
     {#if error}<p role="alert" class="ui-error">{error} Showing the last loaded hosts. <Button variant="ghost" size="sm" onclick={() => load()}>Retry</Button></p>{/if}
     {#if hosts.length}

src/routes/hosts/[id]/+page.svelte

@@ -74,7 +74,7 @@
 <svelte:head><title>{host?.name ?? 'Host'} · Electric Cherry</title></svelte:head>
 <section class="page"><SectionHeader title={host?.name ?? 'Host'} subtitle="Running instances, readiness, inventory, settings, and credentials.">
     {#snippet crumbs()}<Breadcrumbs items={[{ label: 'Hosts', href: '/hosts' }, { label: host?.name ?? 'Host' }]} />{/snippet}
-    {#snippet actions()}<Button href={contextHref(`/capacity?host=${encodeURIComponent($page.params.id ?? "")}`, $page.url)}>Capacity</Button><Button href={contextHref(`/hosts/operations?host=${encodeURIComponent($page.params.id ?? "")}`, $page.url)}>Operation history</Button>{#if canManage && host && (!cleaning || host.decommission?.retryable)}<details class="host-menu"><summary>More actions</summary><div><Button variant="danger" disabled={busy} onclick={() => deleteOpen = true}>{cleaning ? 'Retry cleanup' : 'Delete host'}</Button></div></details>{/if}{/snippet}
+    {#snippet actions()}<Button href={contextHref(`/capacity?host=${encodeURIComponent($page.params.id ?? "")}`, $page.url)}>Capacity</Button><Button href={contextHref(`/capacity?rebalance=1&rebalanceHost=${encodeURIComponent($page.params.id ?? "")}`, $page.url)}>Rebalance</Button><Button href={contextHref(`/hosts/operations?host=${encodeURIComponent($page.params.id ?? "")}`, $page.url)}>Operation history</Button>{#if canManage && host && (!cleaning || host.decommission?.retryable)}<details class="host-menu"><summary>More actions</summary><div><Button variant="danger" disabled={busy} onclick={() => deleteOpen = true}>{cleaning ? 'Retry cleanup' : 'Delete host'}</Button></div></details>{/if}{/snippet}
 </SectionHeader>
     {#if error}<p class="ui-error" role="alert">{error}</p>{/if}{#if notice}<p role="status">{notice}</p>{/if}
     {#if loading}<p role="status">Loading host…</p>{:else if host}

src/routes/provisioning/+page.svelte

@@ -145,7 +145,7 @@
             const sources = row.sources ?? [];
             if (source === 'manual' && (sources.length || row.triggered_by !== 'ui')) return false;
             if (source === 'system' && (sources.length || row.triggered_by === 'ui')) return false;
-            if (['campaign','beta','maintenance'].includes(source) && !sources.some(item => source === 'maintenance' ? ['maintenance','watchdog'].includes(item.kind) : item.kind === source)) return false;
+            if (['campaign','beta','maintenance','rebalance'].includes(source) && !sources.some(item => source === 'maintenance' ? ['maintenance','watchdog'].includes(item.kind) : item.kind === source)) return false;
         }
         return true;
     }));
@@ -175,7 +175,7 @@
         <label class="search">Server<input type="search" bind:value={search} oninput={() => saveContext(true)} placeholder="Search server or lobby" maxlength="100" /></label>
         <label>Operation<select bind:value={action} onchange={() => saveContext(true)}><option value="all">All operations</option>{#each provisioningActions as value}<option value={value}>{actionLabels[value]}</option>{/each}</select></label>
         <label>Status<select bind:value={historyState} onchange={() => saveContext(true)}><option value="all">All statuses</option><option value="pending">Waiting / queued</option><option value="running">In progress</option><option value="awaiting-health">Verifying health</option><option value="done">Complete</option><option value="attention">Needs attention</option><option value="failed">Failed</option><option value="interrupted">Interrupted</option><option value="warning">Warnings</option><option value="cancelled">Cancelled</option></select></label>
-        <label>Source<select bind:value={source} onchange={() => saveContext(true)}><option value="all">All sources</option><option value="manual">Manual</option><option value="campaign">Campaign</option><option value="beta">Beta</option><option value="maintenance">Maintenance</option><option value="system">System</option></select></label>
+        <label>Source<select bind:value={source} onchange={() => saveContext(true)}><option value="all">All sources</option><option value="manual">Manual</option><option value="campaign">Campaign</option><option value="beta">Beta</option><option value="maintenance">Maintenance</option><option value="rebalance">Rebalance</option><option value="system">System</option></select></label>
         {#if filtered}<Button variant="ghost" onclick={clearFilters}>Clear filters</Button>{/if}
     </div>
     <LifecycleDialog bind:open={lifecycleOpen} />

tests/unit/auth-access.test.ts

@@ -7,7 +7,7 @@ const allows = (role: UserRole, route: string | null, method = 'GET') => roleAll
 
 describe('management route permissions', () => {
     it.each([
-        '/', '/api/fleet', '/capacity', '/api/capacity', '/api/servers', '/servers/[name]', '/api/servers/[name]/scripts',
+        '/', '/api/fleet', '/capacity', '/api/capacity', '/api/rebalancing', '/api/servers', '/servers/[name]', '/api/servers/[name]/scripts',
         '/api/metrics', '/metrics', '/api/matches', '/matches/[id]',
         '/leaderboard/players/[game]/[guid]', '/api/campaigns/[id]', '/campaigns/[id]',
         '/api/provisioning/[id]', '/provisioning/[id]', '/api/provisioning/[id]/events',
@@ -22,6 +22,7 @@ describe('management route permissions', () => {
 
     it.each([
         ['/api/servers', 'POST'], ['/api/servers/[name]', 'DELETE'], ['/api/provisioning', 'POST'],
+        ['/api/rebalancing', 'POST'],
         ['/api/provisioning/[id]/cancel', 'POST'], ['/api/provisioning/[id]/resume', 'POST'],
         ['/api/deployment-collections', 'POST'], ['/api/deployment-collections/preview', 'POST'],
         ['/api/deployment-collections/[id]/cancel', 'POST'],

tests/unit/native-provisioning.test.ts

@@ -11,6 +11,9 @@ vi.mock('../../src/lib/server/integrations/rcon.js', () => ({
 }));
 vi.mock('../../src/lib/server/integrations/plutonium.js', () => ({ checkRelease: vi.fn(async () => {}), freshRelease: () => 5354, probeVersion: vi.fn(async () => 5354) }));
 vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}), kickProvisioning: vi.fn(async () => {}) }));
+vi.mock('../../src/lib/server/rebalancing/admission.js', () => ({
+    rebalanceAdmission: vi.fn(() => null), requestRebalanceStopForOperation: vi.fn()
+}));
 
 let queuedKeys = 0;
 let fixture: Fixture; let hostId: string; let libraryId: string;
@@ -84,6 +87,123 @@ async function observations(server: string) {
     return [serverObservation(row), observationReader()(row), new MonitoringSnapshot().observe(row)];
 }
 
+async function moveTarget() {
+    const hosts = await import('../../src/lib/server/hosts/index.js');
+    const libraries = await import('../../src/lib/server/game-files/index.js');
+    const target = hosts.createHost({ name: 'Rebalance destination', portMin: 29000, portMax: 29005 });
+    hosts.enrollHost(hosts.createEnrollment(target.id).token, inventory());
+    libraries.updateHostLibraryCaches(target.id, [{ libraryVersionId: libraryId, status: 'ready',
+        manifestSha256: libraries.getLibraryVersion(libraryId)!.manifestSha256 }]);
+    return target;
+}
+
+describe('rebalance instance migrations', () => {
+    it('preserves deployed configuration, patched file versions, identity and credentials on an active source', async () => {
+        const { config } = await import('../../src/lib/server/config.js');
+        Object.assign(config.rcon, { password: '' });
+        const { createScript, uploadNewVersion } = await import('../../src/lib/server/fleet/scripts.js');
+        const { db } = await import('../../src/lib/server/db/index.js');
+        const baseline = createScript({ name: 'Deployed baseline', kind: 'baseline', scopeType: 'game', scopeValue: 't5',
+            content: Buffer.from('set sv_voice "1"\n') });
+        const script = createScript({ name: 'Deployed script', kind: 'script', targetRelPath: 'preserved.gsc',
+            scopeType: 'game', scopeValue: 't5', content: Buffer.from('original deployed script') });
+        for (const artifact of [baseline, script]) db().prepare('UPDATE scripts SET current_version_id=? WHERE id=?').run(artifact.version.id, artifact.script.id);
+        const job = await queue(); await advance();
+        const launch = (await operationFor(job.deploymentId))!.payload as { rconPassword: string; files: { path: string; sha256: string }[] };
+        await acknowledge(job.deploymentId);
+        const patched = uploadNewVersion(script.script.id, Buffer.from('confirmed patch after initial launch'));
+        const { recordScript } = await import('../../src/lib/server/fleet/inventory.js');
+        recordScript(job.name, 'preserved.gsc', patched.id, patched.sha256, patched.size, 'campaign');
+        const newest = uploadNewVersion(script.script.id, Buffer.from('new promotion must not be installed'));
+        db().prepare('UPDATE scripts SET current_version_id=? WHERE id=?').run(newest.id, script.script.id);
+        const newBaseline = uploadNewVersion(baseline.script.id, Buffer.from('set sv_voice "0"\n'));
+        db().prepare('UPDATE scripts SET current_version_id=? WHERE id=?').run(newBaseline.id, baseline.script.id);
+        Object.assign(config.rcon, { password: 'new-controller-password' });
+        const target = await moveTarget();
+        const { captureMoveRecipe, requestInstanceMove, hostMigration } = await import('../../src/lib/server/provisioning/host-drain.js');
+        const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
+        const original = nativeInstance(job.name)!;
+        const recipe = captureMoveRecipe(job.name);
+        expect(recipe.files.every(file => file.content === undefined)).toBe(true);
+        const id = requestInstanceMove(job.name, target.id, { triggeredBy: 'rebalance:test', recipe });
+        await advance();
+        const removal = hostMigration(id)!.remove_operation_id!;
+        expect(removal).toBeTruthy();
+        const { cancelProvisioningJob, canCancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
+        expect(canCancelProvisioningJob(id)).toBe(false);
+        expect(cancelProvisioningJob(id)).toBe(false);
+        const { getDeployment, listDeployments } = await import('../../src/lib/server/provisioning/history.js');
+        expect(getDeployment(id)?.canCancel).toBe(false);
+        expect(listDeployments().find(item => item.id === id)?.canCancel).toBe(0);
+        const { POST } = await import('../../src/routes/api/deployments/[id]/cancel/+server.js');
+        const response = POST({ params: { id } });
+        expect(response.status).toBe(409);
+        expect(await response.json()).toMatchObject({ error: expect.stringContaining('must finish recreating') });
+        const { reportHostOperation, getHost } = await import('../../src/lib/server/hosts/index.js');
+        reportHostOperation(hostId, removal, { state: 'completed' });
+        await advance(); await advance();
+        const creation = (await operationFor(id))!;
+        expect(creation).toMatchObject({ hostId: target.id, kind: 'instance.create', generation: 2,
+            payload: { name: job.name, libraryId, image, rconPassword: launch.rconPassword, env: { PLUTO_SERVER_KEY: 'saved-key' } } });
+        const payload = creation.payload as { files: { path: string; sha256: string }[] };
+        expect(payload.files).toContainEqual(expect.objectContaining({ path: expect.stringContaining('preserved.gsc'), sha256: patched.sha256 }));
+        expect(payload.files.find(file => file.path === 'configs/dedicated_sp.cfg')?.sha256)
+            .toBe(launch.files.find(file => file.path === 'configs/dedicated_sp.cfg')?.sha256);
+        reportHostOperation(target.id, creation.id, { state: 'completed', result: { state: 'running', healthy: true } });
+        await advance();
+        expect((await detail(id)).state).toBe('done');
+        expect(nativeInstance(job.name)).toMatchObject({ id: original.id, host_id: target.id, generation: 2 });
+        expect(getHost(hostId)?.mode).toBe('active');
+        expect(JSON.stringify(getDeployment(id)?.recipe)).not.toContain(launch.rconPassword);
+    });
+
+    it('rechecks occupancy, target capacity and run admission before removing the source', async () => {
+        const job = await queue(); await advance(); await acknowledge(job.deploymentId);
+        const target = await moveTarget();
+        const { requestInstanceMove, hostMigration } = await import('../../src/lib/server/provisioning/host-drain.js');
+        expect(() => requestInstanceMove(job.name, hostId, { triggeredBy: 'rebalance:test' })).toThrow('different destination');
+        const id = requestInstanceMove(job.name, target.id, { triggeredBy: 'rebalance:test' });
+        remote.players = 2; await advance();
+        expect(hostMigration(id)?.remove_operation_id).toBeNull();
+        remote.players = 0;
+        const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
+        heartbeat(target.id, { ...inventory(), memoryAvailable: 1 });
+        await advance();
+        expect(hostMigration(id)?.remove_operation_id).toBeNull();
+        heartbeat(target.id, inventory());
+        const { rebalanceAdmission } = await import('../../src/lib/server/rebalancing/admission.js');
+        vi.mocked(rebalanceAdmission).mockReturnValueOnce('Run stopped during occupancy check.');
+        await advance();
+        expect(hostMigration(id)?.remove_operation_id).toBeNull();
+        await advance();
+        expect(hostMigration(id)?.remove_operation_id).toEqual(expect.any(String));
+    });
+
+    it('withdraws only uncommitted moves and preserves their server keys', async () => {
+        const job = await queue(); await advance(); await acknowledge(job.deploymentId);
+        const target = await moveTarget();
+        const { requestInstanceMove } = await import('../../src/lib/server/provisioning/host-drain.js');
+        const id = requestInstanceMove(job.name, target.id, { triggeredBy: 'rebalance:test' });
+        const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
+        expect(cancelProvisioningJob(id)).toBe(true);
+        const { requestRebalanceStopForOperation } = await import('../../src/lib/server/rebalancing/admission.js');
+        expect(requestRebalanceStopForOperation).toHaveBeenCalledWith(id);
+        await advance();
+        const { getServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
+        expect(getServerKey(job.name)).toBe('saved-key');
+        const { db } = await import('../../src/lib/server/db/index.js');
+        expect(db().prepare("SELECT id FROM ec_host_operations WHERE kind='instance.remove'").all()).toEqual([]);
+    });
+
+    it('refuses to move when recorded deployed bytes cannot be restored', async () => {
+        const job = await queue(); await advance(); await acknowledge(job.deploymentId);
+        const { captureMoveRecipe } = await import('../../src/lib/server/provisioning/host-drain.js');
+        const { db } = await import('../../src/lib/server/db/index.js');
+        db().prepare("UPDATE server_scripts SET sha256=? WHERE server=?").run('0'.repeat(64), job.name);
+        expect(() => captureMoveRecipe(job.name)).toThrow('deployed bytes are unavailable');
+    });
+});
+
 describe('runtime configuration at dispatch', () => {
     it.each([
         ['t5', 'shared'], ['t5', 'generated'], ['t6', 'shared'], ['t6', 'generated']

tests/unit/provisioning-list.test.ts

@@ -70,6 +70,69 @@ describe('bounded provisioning lists', () => {
         expect(JSON.stringify(deploymentListPage({ view: 'active' }))).not.toContain('private snapshot');
     });
 
+    it('puts verification and executing work ahead of preflight waits and capacity queues', async () => {
+        const { add, handle, listDeployments, deploymentListPage } = await setup();
+        const addProgress = (id: string, state: string, second: number, phase: string,
+            waitCode: string | null = null, stage = 'Recorded', action = 'rebuild') => {
+            add(id, state, action, 't5_zombie_theater_1', null, stage, `2026-01-01T00:00:${String(second).padStart(2, '0')}.000Z`);
+            handle.prepare('UPDATE server_deployments SET phase_key=?, observation_json=? WHERE id=?')
+                .run(phase, JSON.stringify({ blocker: waitCode ? { code: waitCode } : null }), id);
+        };
+        // Request order is deliberately different from progress order.
+        addProgress('queued', 'queued', 1, 'requested');
+        addProgress('cleanup', 'waiting', 12, 'cleanup', null, 'Removing server files and registry', 'destroy');
+        addProgress('verify-newer', 'awaiting-health', 21, 'verify-map');
+        addProgress('capacity', 'running', 2, 'startup-queue', 'startup-capacity');
+        addProgress('operation-conflict', 'running', 6, 'container-action', 'operation-conflict');
+        addProgress('agent-wait', 'running', 11, 'container-action', 'native-host', 'Agent operation pending');
+        addProgress('host-wait', 'waiting', 5, 'eligibility', 'native-host', 'Host is unavailable; waiting for reconnection.');
+        addProgress('staging', 'waiting', 7, 'eligibility', 'native-host', 'Staging and verifying files on the host.');
+        addProgress('staging-wait', 'waiting', 8, 'eligibility', 'native-host', 'Waiting for verified remote file staging.');
+        addProgress('legacy-capacity', 'waiting', 3, 'eligibility', 'native-host', 'Queued to rebuild; waiting for another server to recover');
+        addProgress('verify-older', 'awaiting-health', 20, 'verify-container', 'native-host', 'Waiting for container readiness: starting');
+        addProgress('preparing-queued', 'queued', 9, 'prepare');
+        addProgress('executing', 'running', 10, 'prepare');
+        addProgress('eligibility', 'running', 4, 'eligibility', 'eligibility');
+        addProgress('interrupted', 'interrupted', 30, 'container-action', 'native-host');
+        addProgress('failed', 'failed', 31, 'verify-map', 'native-host');
+
+        const expected = ['verify-older', 'verify-newer', 'staging', 'staging-wait', 'preparing-queued', 'executing', 'agent-wait', 'cleanup',
+            'eligibility', 'host-wait', 'operation-conflict', 'queued', 'capacity', 'legacy-capacity'];
+        expect(listDeployments({ view: 'active' }).map(row => row.id)).toEqual(expected);
+        const { GET } = await import('../../src/routes/api/provisioning/+server.js');
+        const active = await GET({ url: new URL('http://localhost/api/provisioning?view=active&limit=1') }).json();
+        expect(active.deployments.map((row: { id: string }) => row.id)).toEqual(expected);
+        expect(active).toMatchObject({ total: expected.length, needsReview: 2 });
+
+        // Progress priority applies only to Active; retained outcomes and mixed reads stay newest first.
+        expect(deploymentListPage({ view: 'history', limit: 1 }).deployments.map(row => row.id)).toEqual(['failed']);
+        expect(deploymentListPage({ view: 'history', limit: 1, offset: 1 }).deployments.map(row => row.id)).toEqual(['interrupted']);
+        expect(listDeployments({ view: 'all', limit: 3 }).map(row => row.id)).toEqual(['failed', 'interrupted', 'verify-newer']);
+    });
+
+    it('keeps oldest requests and insertion ties stable while active progress is polled', async () => {
+        const { add, handle, listDeployments } = await setup();
+        add('z-first', 'awaiting-health', 'rebuild', 't5_zombie_theater_1', null, 'Checking game status', '2026-01-02');
+        add('a-second', 'awaiting-health', 'rebuild', 't5_zombie_theater_1', null, 'Checking selected engine map', '2026-01-02');
+        add('old-inserted-last', 'awaiting-health', 'rebuild', 't5_zombie_theater_1', null, 'Checking container state', '2026-01-01');
+        add('queued', 'queued', 'rebuild', 't5_zombie_theater_1', null, 'Request recorded', '2025-12-31');
+        const ids = () => listDeployments({ view: 'active' }).map(row => row.id);
+        const expected = ['old-inserted-last', 'z-first', 'a-second', 'queued'];
+        expect(ids()).toEqual(expected);
+        expect(ids()).toEqual(expected);
+
+        handle.prepare(`UPDATE server_deployments SET phase_key='verify-container', phase_started_at='2026-01-03',
+            observed_at='2026-01-03', observation_json=? WHERE id='z-first'`)
+            .run(JSON.stringify({ blocker: { code: 'native-host', message: 'Waiting for container readiness: starting' } }));
+        handle.prepare("UPDATE server_deployments SET phase_key='verify-plutonium', phase_started_at='2026-01-04' WHERE id='a-second'").run();
+        expect(ids()).toEqual(expected);
+
+        handle.prepare("UPDATE server_deployments SET state='done', finished_at='2026-01-05' WHERE id='z-first'").run();
+        handle.prepare("UPDATE server_deployments SET state='awaiting-health', phase_key='verify-game' WHERE id='queued'").run();
+        expect(ids()).toEqual(['queued', 'old-inserted-last', 'a-second']);
+        expect(listDeployments({ view: 'all' }).map(row => row.id)).toEqual(['a-second', 'z-first', 'old-inserted-last', 'queued']);
+    });
+
     it('polls every active row and counts review outcomes without transferring retained history', async () => {
         const { handle, add, REMOVING_SERVER } = await setup();
         handle.transaction(() => {

tests/unit/provisioning-presentation.test.ts

@@ -163,6 +163,18 @@ describe('provisioning response and return-context boundaries', () => {
         expect(provisioningReturn('/collections/launches/launch-a').href).toBe('/collections/launches/launch-a');
     });
 
+    it('returns rebalance moves to their run while discarding unrelated navigation parameters', () => {
+        expect(provisioningReturn('/capacity?rebalance=1&rebalanceRun=run-a&redirect=https://evil.test'))
+            .toEqual({ href: '/capacity?rebalance=1&rebalanceRun=run-a', label: 'Back to rebalance' });
+        expect(provisioningReturn('/capacity?rebalance=1')).toEqual({ href: '/capacity?rebalance=1', label: 'Back to rebalance' });
+        expect(provisioningReturn('/capacity?rebalanceRun=run-a').href).toBe('/provisioning');
+        const detail = fixture();
+        detail.sources = [{ kind: 'rebalance', source_id: 'run-a', label: 'Rebalance', href: '/capacity?rebalance=1&rebalanceRun=run-a' }];
+        expect(isProvisioningDetail(detail, detail.deployment.id)).toBe(true);
+        detail.sources[0].href = '//evil.test';
+        expect(isProvisioningDetail(detail, detail.deployment.id)).toBe(false);
+    });
+
     it('rejects incomplete nested snapshots before they replace a working detail', () => {
         const detail = fixture();
         expect(isProvisioningDetail(detail, detail.deployment.id)).toBe(true);

tests/unit/rebalancing-planner.test.ts

@@ -0,0 +1,92 @@
+import { describe, expect, it } from 'vitest';
+import { rebalanceSettings, type RebalanceSettings } from '../../src/lib/rebalancing.js';
+import { planRebalance, rebalanceMoveBlocker, type RebalancingHost, type RebalancingInstance, type RebalancingSnapshot } from '../../src/lib/server/rebalancing/planner.js';
+
+const gib = 1024 ** 3;
+const settings: RebalanceSettings = { hostIds: ['a', 'b'], game: 'all', maxMoves: 20 };
+const host = (id: string, cpu: number, count = 0): RebalancingHost => ({ id, name: id, cpuCount: 8, memoryTotal: 16 * gib,
+    used: { cpuCores: cpu, memoryBytes: gib }, running: { cpuCores: 0.5, memoryBytes: gib },
+    startup: { cpuCores: 1, memoryBytes: 2 * gib }, instances: count, ports: 20, reason: null });
+const instance = (n: number, source = 'a', cpu = 1): RebalancingInstance => ({ server: `t6_zombie_town_${n}`, instanceId: `i${n}`, generation: 1,
+    sourceHostId: source, game: 't6', relief: { cpuCores: cpu, memoryBytes: gib }, demand: { cpuCores: cpu, memoryBytes: gib }, destinations: { a: null, b: null } });
+const snapshot = (hosts = [host('a', 6, 6), host('b', 0)], instances = Array.from({ length: 6 }, (_, n) => instance(n))): RebalancingSnapshot => ({ hosts, instances, exclusions: [], warnings: [] });
+
+describe('rebalance planning', () => {
+    it('validates bounded hosts and moves without silently choosing all hosts', () => {
+        expect(rebalanceSettings({ ...settings, hostIds: ['b', 'a'] }).hostIds).toEqual(['a', 'b']);
+        for (const input of [{}, { ...settings, hostIds: ['a'] }, { ...settings, hostIds: ['a', 'a'] }, { ...settings, game: 't4' },
+            { ...settings, maxMoves: 0 }, { ...settings, maxMoves: 21 }, { ...settings, maxMoves: 2.5 }]) expect(() => rebalanceSettings(input)).toThrow();
+    });
+
+    it('spreads pressure with the fewest useful moves and never mutates live measurements', () => {
+        const input = snapshot(), saved = structuredClone(input);
+        const result = planRebalance(input, settings);
+        expect(result.moves.map(move => [move.server, move.sourceHostId, move.targetHostId])).toEqual([0, 1, 2].map(n => [`t6_zombie_town_${n}`, 'a', 'b']));
+        expect(result.hosts.map(item => item.after.instances)).toEqual([3, 3]);
+        expect(result.hosts.map(item => item.after.cpuPercent)).toEqual([37.5, 37.5]);
+        expect(new Set(result.moves.map(move => move.instanceId)).size).toBe(result.moves.length);
+        expect(input).toEqual(saved);
+    });
+
+    it('uses host size and both resource dimensions rather than equal counts', () => {
+        const small = host('a', 3, 3); small.cpuCount = 4;
+        const large = host('b', 4, 4); large.cpuCount = 16;
+        expect(planRebalance(snapshot([small, large], [instance(1)]), settings).moves).toHaveLength(1);
+        small.used = { cpuCores: 1, memoryBytes: 12 * gib }; large.used = { cpuCores: 1, memoryBytes: 2 * gib };
+        const memory = instance(1); memory.relief.memoryBytes = 2 * gib; memory.demand.memoryBytes = 2 * gib;
+        expect(planRebalance(snapshot([small, large], [memory]), settings).moves[0].improvement).toBeCloseTo(12.5);
+    });
+
+    it('respects startup fit, port capacity, compatibility, host eligibility, and reservations', () => {
+        for (const mutate of [
+            (target: RebalancingHost) => { target.ports = 0; },
+            (target: RebalancingHost) => { target.reason = 'Host disabled.'; },
+            (target: RebalancingHost) => { target.startup.cpuCores = 8; },
+            (target: RebalancingHost) => { target.startup.memoryBytes = 16 * gib; },
+            (target: RebalancingHost) => { target.used.cpuCores = 7; }
+        ]) {
+            const input = snapshot(); mutate(input.hosts[1]);
+            expect(planRebalance(input, settings).moves).toEqual([]);
+        }
+        const input = snapshot(); input.instances.forEach(item => { item.destinations.b = 'Missing the deployed runtime.'; });
+        expect(planRebalance(input, settings).moves).toEqual([]);
+        expect(planRebalance(input, settings).exclusions[0].reason).toContain('Missing the deployed runtime');
+        input.instances.forEach(item => { delete item.destinations.b; });
+        expect(planRebalance(input, settings).moves).toEqual([]);
+    });
+
+    it('requires five percentage points of improvement, preserves limits, and does not move twice', () => {
+        const input = snapshot([host('a', 4), host('b', 2)], [instance(1, 'a', 0.3)]);
+        expect(planRebalance(input, settings).moves).toEqual([]);
+        input.instances[0].relief.cpuCores = 0.4; input.instances[0].demand.cpuCores = 0.4;
+        expect(planRebalance(input, settings).moves[0].improvement).toBeCloseTo(5);
+        expect(planRebalance(snapshot(), { ...settings, maxMoves: 1 }).moves).toHaveLength(1);
+        expect(planRebalance(snapshot(), { ...settings, game: 't5' }).moves).toHaveLength(0);
+        const onePort = snapshot(); onePort.hosts[1].ports = 1;
+        expect(planRebalance(onePort, settings).moves).toHaveLength(1);
+    });
+
+    it('prefers the largest improvement deterministically and revalidates the approved pair', () => {
+        const input = snapshot([host('a', 6, 6), host('b', 0)], [instance(3), instance(2, 'a', 2), instance(1)]);
+        const move = planRebalance(input, { ...settings, maxMoves: 1 }).moves[0];
+        expect(move.server).toBe('t6_zombie_town_2');
+        expect(rebalanceMoveBlocker(input, move)).toBeNull();
+        expect(rebalanceMoveBlocker(input, { ...move, generation: 2 })).toContain('generation has changed');
+        input.hosts[1].used.cpuCores = 6;
+        expect(rebalanceMoveBlocker(input, move)).toBeTruthy();
+        input.instances = []; input.exclusions.push({ server: move.server, reason: 'Players are present.' });
+        expect(rebalanceMoveBlocker(input, move)).toBe('Players are present.');
+    });
+
+    it('only credits source ports actually released inside the configured range and bind', () => {
+        const a = host('a', 6.8, 1), b = host('b', 5.2, 1), c = host('c', 0);
+        a.ports = 0; b.ports = 0; c.ports = 1;
+        const first = instance(1, 'a', 4), second = instance(2, 'b', 1);
+        first.destinations.c = null; second.destinations.c = null;
+        const input = snapshot([a, b, c], [first, second]);
+        const selection = { ...settings, hostIds: ['a', 'b', 'c'] };
+        expect(planRebalance(input, selection).moves).toHaveLength(1);
+        first.releasesPort = true;
+        expect(planRebalance(input, selection).moves).toHaveLength(2);
+    });
+});

tests/unit/rebalancing-runs.test.ts

@@ -0,0 +1,299 @@
+import { randomUUID } from 'node:crypto';
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+import { newFixture, type Fixture } from '../helpers/fixture.js';
+import { prepareNativeHost } from '../helpers/native-host.js';
+import type { RebalanceMove } from '../../src/lib/rebalancing.js';
+
+vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}), kickProvisioning: vi.fn(async () => {}) }));
+vi.mock('../../src/lib/server/integrations/plutonium.js', () => ({ checkRelease: vi.fn(async () => {}), freshRelease: () => 5354, probeVersion: vi.fn(async () => 5354) }));
+vi.mock('../../src/lib/server/integrations/rcon.js', () => ({
+    rconStatusResult: vi.fn(async () => ({ ok: true, endpoint: { host: 'agent', port: 28960 }, rawReply: '',
+        status: { players: 0, map: 'zombie_theater', gametype: 'zom', roster: [] } })),
+    rconGetPlayerCount: vi.fn(async () => 0), rconPlutoniumVersion: vi.fn(async () => ({ revision: 5354, error: null }))
+}));
+
+let fixture: Fixture, sourceId: string, targetId: string, libraryId: string;
+const names = Array.from({ length: 4 }, (_, index) => `t5_zombie_theater_${index + 1}`);
+const image = 'example.test/plutainer:rebalance';
+const gib = 1024 ** 3;
+const clockStep = () => vi.setSystemTime(Date.now() + 1000);
+const settings = () => ({ hostIds: [sourceId, targetId], game: 'all' as const, maxMoves: 5 });
+
+async function heartbeat(hostId: string, cpuUsagePercent: number) {
+    const { db } = await import('../../src/lib/server/db/index.js');
+    const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
+    const instances = db().prepare('SELECT id,generation FROM native_instances WHERE host_id=? AND retired_at IS NULL')
+        .all(hostId) as { id: string; generation: number }[];
+    heartbeat(hostId, { protocolVersion: 1, cpuCount: 16, cpuUsagePercent, memoryTotal: 32 * gib,
+        memoryAvailable: 32 * gib - (2 + instances.length * 2) * gib, dockerRunning: true,
+        directoriesWritable: true, storageAvailable: 100 * gib, runtimeImages: [image],
+        libraries: [{ id: libraryId, state: 'ready' }], instanceResources: instances.map(instance => ({
+            instanceId: instance.id, generation: instance.generation, uptimeSeconds: 3600, cpuCores: 3, memoryBytes: 2 * gib })) });
+}
+
+beforeEach(async () => {
+    vi.resetModules(); vi.clearAllMocks();
+    vi.useFakeTimers({ toFake: ['Date'] }); vi.setSystemTime(new Date('2026-10-08T16:00:00.000Z'));
+    fixture = await newFixture([]); vi.stubEnv('PLUTAINER_IMAGE', image); vi.stubEnv('RCON_PASSWORD', 'preserved-password');
+    const { db } = await import('../../src/lib/server/db/index.js');
+    const at = new Date().toISOString(), launched = new Date(Date.now() - 600_000).toISOString();
+    for (const [index, name] of names.entries()) db().prepare(`INSERT INTO servers(name,game,map,game_mode,map_slug,
+        config_file,game_id,image,game_port,container_state,container_checked_at,last_players,last_map,last_seen_at,last_checked_at,last_roster)
+        VALUES(?,'t5','zombie_theater','zombie','theater','dedicated_sp.cfg','t5sp',?,?,'running',?,0,'zombie_theater',?,?,'[]')`)
+        .run(name, image, 28960 + index, at, at, at);
+    const prepared = await prepareNativeHost(names); sourceId = prepared.host.id; libraryId = prepared.versions.get('t5')!;
+    const hosts = await import('../../src/lib/server/hosts/index.js');
+    targetId = hosts.createHost({ name: 'Empty destination', portMin: 29000, portMax: 29020 }).id;
+    hosts.enrollHost(hosts.createEnrollment(targetId).token, { protocolVersion: 1, cpuCount: 16, cpuUsagePercent: 6.25,
+        memoryTotal: 32 * gib, memoryAvailable: 30 * gib, dockerRunning: true, directoriesWritable: true,
+        storageAvailable: 100 * gib, runtimeImages: [image], libraries: [{ id: libraryId, state: 'ready' }] });
+    const libraries = await import('../../src/lib/server/game-files/index.js');
+    libraries.updateHostLibraryCaches(targetId, [{ libraryVersionId: libraryId, status: 'ready',
+        manifestSha256: libraries.getLibraryVersion(libraryId)!.manifestSha256 }]);
+    const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
+    const { beginDeployment } = await import('../../src/lib/server/provisioning/history.js');
+    const { captureRecipe, recipeStorage } = await import('../../src/lib/server/fleet/launch-recipes.js');
+    const { reserveServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
+    const { recordScript } = await import('../../src/lib/server/fleet/inventory.js');
+    for (const name of names) {
+        const instance = nativeInstance(name)!;
+        reserveServerKey(name, 't5', `saved-key-${name}`);
+        for (const protocol of ['udp', 'tcp'] as const) hosts.reserveHostPort(sourceId, instance.id, { port: instance.port, bindAddress: '0.0.0.0', protocol });
+        const id = beginDeployment(name, 'create', 'fixture');
+        const operation = hosts.queueHostOperation(sourceId, 'instance.create', { name, rconPassword: 'preserved-password' },
+            { instanceId: instance.id, generation: 1, idempotencyKey: randomUUID() });
+        hosts.reportHostOperation(sourceId, operation.id, { state: 'completed', result: { state: 'running', healthy: true } });
+        db().prepare("UPDATE server_deployments SET state='done',submitted_at=?,finished_at=? WHERE id=?").run(launched, launched, id);
+        db().prepare('UPDATE native_provisioning SET operation_id=?,verified_at=? WHERE deployment_id=?').run(operation.id, launched, id);
+        const recipe = captureRecipe('t5', 'theater', 'preserved-password');
+        const stored = recipeStorage({ ...recipe, image, native: { hostId: sourceId, libraryId, image } });
+        db().prepare('INSERT INTO server_launch_recipes(deployment_id,recipe,recipe_summary) VALUES(?,?,?)').run(id, stored.recipe, stored.recipeSummary);
+        for (const file of recipe.files) recordScript(name, file.path, file.versionId, file.sha256, file.size, 'native-create');
+    }
+    await heartbeat(sourceId, 75); await heartbeat(targetId, 6.25);
+    const backend = await import('../../src/lib/server/backends/native.js');
+    vi.spyOn(backend, 'nativeRequest').mockImplementation(async (_instance, kind) => {
+        if (kind !== 'instance.inspect') throw new Error(`Unexpected host request ${kind}`);
+        return { state: 'running', healthy: true, exists: true } as never;
+    });
+});
+afterEach(async () => { vi.useRealTimers(); vi.restoreAllMocks(); await fixture.cleanup(); vi.unstubAllEnvs(); });
+
+async function start() {
+    const service = await import('../../src/lib/server/rebalancing/index.js');
+    const preview = service.previewRebalance(settings());
+    expect(preview.moves.length, JSON.stringify(preview.exclusions)).toBeGreaterThanOrEqual(2);
+    return service.startRebalance(preview.id);
+}
+async function currentMove(runId: string): Promise<RebalanceMove> {
+    const { getRebalanceRun } = await import('../../src/lib/server/rebalancing/index.js');
+    return getRebalanceRun(runId)!.moves.find(move => move.state === 'moving')!;
+}
+async function provision() {
+    const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
+    await processProvisioning();
+}
+async function finishMove(move: RebalanceMove) {
+    const { hostMigration } = await import('../../src/lib/server/provisioning/host-drain.js');
+    const hosts = await import('../../src/lib/server/hosts/index.js');
+    const { db } = await import('../../src/lib/server/db/index.js');
+    const removal = hostMigration(move.operationId!)!.remove_operation_id;
+    expect(removal).toBeTruthy();
+    clockStep(); hosts.reportHostOperation(sourceId, removal!, { state: 'completed' });
+    await provision(); await provision();
+    const created = db().prepare('SELECT operation_id FROM native_provisioning WHERE deployment_id=?').get(move.operationId) as { operation_id: string | null };
+    expect(created.operation_id).toBeTruthy();
+    clockStep(); hosts.reportHostOperation(targetId, created.operation_id!, { state: 'completed', result: { state: 'running', healthy: true } });
+    clockStep(); await provision();
+    expect(db().prepare('SELECT state FROM server_deployments WHERE id=?').get(move.operationId)).toEqual({ state: 'done' });
+}
+
+describe('manual rebalance runs', () => {
+    it('previews without reserving capacity and rejects stale plans', async () => {
+        const { db } = await import('../../src/lib/server/db/index.js');
+        const { previewRebalance, startRebalance } = await import('../../src/lib/server/rebalancing/index.js');
+        const ports = db().prepare('SELECT * FROM ec_host_ports').all();
+        const commands = db().prepare('SELECT id FROM ec_host_operations').all();
+        const preview = previewRebalance(settings());
+        expect(preview.moves.length).toBeGreaterThanOrEqual(2);
+        expect(db().prepare('SELECT * FROM ec_host_ports').all()).toEqual(ports);
+        expect(db().prepare('SELECT id FROM ec_host_operations').all()).toEqual(commands);
+        expect(db().prepare('SELECT * FROM rebalancing_runs').all()).toEqual([]);
+        db().prepare("UPDATE servers SET desired_state='stopped' WHERE name=?").run(preview.moves[0].server);
+        expect(() => startRebalance(preview.id)).toThrow('fleet changed');
+    });
+
+    it('starts idempotently, rejects concurrent runs, and queues exactly one move', async () => {
+        const service = await import('../../src/lib/server/rebalancing/index.js');
+        const preview = service.previewRebalance(settings());
+        const competing = service.previewRebalance(settings());
+        const run = service.startRebalance(preview.id);
+        expect(service.startRebalance(preview.id).id).toBe(run.id);
+        expect(() => service.startRebalance(competing.id)).toThrow('current rebalance');
+        expect(await service.runRebalancing()).toBe(true);
+        expect(await service.runRebalancing()).toBe(false);
+        const moves = service.getRebalanceRun(run.id)!.moves;
+        expect(moves.filter(move => move.operationId)).toHaveLength(1);
+        expect(moves.filter(move => move.state === 'planned').length).toBeGreaterThan(0);
+    });
+
+    it('rejects expired previews but returns an already-started run on a late retry', async () => {
+        const service = await import('../../src/lib/server/rebalancing/index.js');
+        const first = service.previewRebalance(settings());
+        const expired = service.previewRebalance(settings());
+        const run = service.startRebalance(first.id);
+        vi.setSystemTime(Date.now() + 6 * 60_000);
+        expect(service.startRebalance(first.id).id).toBe(run.id);
+        expect(() => service.startRebalance(expired.id)).toThrow('Preview expired');
+    });
+
+    it('stops before removal without changing placement or releasing the key, and allows a later host drain', async () => {
+        const run = await start();
+        const service = await import('../../src/lib/server/rebalancing/index.js');
+        await service.runRebalancing();
+        const move = await currentMove(run.id);
+        const stopped = service.stopRebalance(run.id);
+        expect(stopped.state).toBe('stopped');
+        expect(stopped.moves.every(move => move.state === 'skipped')).toBe(true);
+        await provision();
+        const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
+        const { getServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
+        expect(nativeInstance(move.server)?.host_id).toBe(sourceId);
+        expect(getServerKey(move.server)).toBe(`saved-key-${move.server}`);
+        const { db } = await import('../../src/lib/server/db/index.js');
+        expect(db().prepare("SELECT id FROM ec_host_operations WHERE kind='instance.remove'").all()).toEqual([]);
+        const { updateHost } = await import('../../src/lib/server/hosts/index.js');
+        updateHost(sourceId, { mode: 'draining' });
+        const { reconcileHostDrains } = await import('../../src/lib/server/provisioning/host-drain.js');
+        reconcileHostDrains();
+        expect(db().prepare("SELECT id FROM server_deployments WHERE server=? AND triggered_by='host-drain'").get(move.server)).toBeTruthy();
+    });
+
+    it('finishes a committed move after Stop while withdrawing the remaining moves', async () => {
+        const run = await start();
+        const service = await import('../../src/lib/server/rebalancing/index.js');
+        await service.runRebalancing(); await provision();
+        const move = await currentMove(run.id);
+        expect(service.stopRebalance(run.id).state).toBe('stopping');
+        const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
+        expect(cancelProvisioningJob(move.operationId!)).toBe(false);
+        await finishMove(move); await service.runRebalancing();
+        const finished = service.getRebalanceRun(run.id)!;
+        expect(finished.state).toBe('stopped');
+        expect(finished.moves.filter(move => move.state === 'done')).toHaveLength(1);
+        expect(finished.moves.filter(move => move.state === 'skipped')).toHaveLength(run.moves.length - 1);
+        const { nativeInstance } = await import('../../src/lib/server/backends/native.js');
+        const { getHost } = await import('../../src/lib/server/hosts/index.js');
+        expect(nativeInstance(move.server)).toMatchObject({ id: move.instanceId, generation: 2, host_id: targetId });
+        expect(getHost(sourceId)?.mode).toBe('active');
+    });
+
+    it('treats cancellation from Provisioning as stopping the rest of the run', async () => {
+        const run = await start();
+        const service = await import('../../src/lib/server/rebalancing/index.js');
+        await service.runRebalancing();
+        const move = await currentMove(run.id);
+        const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
+        expect(cancelProvisioningJob(move.operationId!)).toBe(true);
+        expect(service.getRebalanceRun(run.id)?.state).toBe('stopping');
+        await service.runRebalancing();
+        expect(service.getRebalanceRun(run.id)?.state).toBe('stopped');
+        expect(service.getRebalanceRun(run.id)?.moves.every(move => move.state === 'skipped')).toBe(true);
+    });
+
+    it('waits for both hosts to report fresh measurements before admitting the next move', async () => {
+        const run = await start();
+        const service = await import('../../src/lib/server/rebalancing/index.js');
+        await service.runRebalancing(); await provision();
+        await finishMove(await currentMove(run.id));
+        expect(await service.runRebalancing()).toBe(false);
+        expect(service.getRebalanceRun(run.id)?.message).toContain('fresh source and destination');
+        expect(service.getRebalanceRun(run.id)?.moves.filter(move => move.operationId)).toHaveLength(1);
+        clockStep(); await heartbeat(sourceId, 56.25);
+        expect(await service.runRebalancing()).toBe(false);
+        await heartbeat(targetId, 25);
+        expect(await service.runRebalancing()).toBe(true);
+        expect(service.getRebalanceRun(run.id)?.moves.filter(move => move.operationId)).toHaveLength(2);
+    });
+
+    it('skips an uncommitted move that becomes occupied and continues other approved work', async () => {
+        const run = await start();
+        const service = await import('../../src/lib/server/rebalancing/index.js');
+        await service.runRebalancing();
+        const move = await currentMove(run.id);
+        const { db } = await import('../../src/lib/server/db/index.js');
+        db().prepare('UPDATE servers SET last_players=2 WHERE name=?').run(move.server);
+        await service.runRebalancing();
+        const fresh = service.getRebalanceRun(run.id)!;
+        expect(fresh.state).toBe('running');
+        expect(fresh.moves.find(item => item.id === move.id)?.state).toBe('skipped');
+        expect(fresh.moves.filter(item => item.state === 'moving')).toHaveLength(1);
+    });
+
+    it('halts remaining moves after a provisioning failure', async () => {
+        const run = await start();
+        const service = await import('../../src/lib/server/rebalancing/index.js');
+        await service.runRebalancing();
+        const move = await currentMove(run.id);
+        const { db } = await import('../../src/lib/server/db/index.js');
+        db().prepare("UPDATE server_deployments SET state='failed',error='Recovery needed' WHERE id=?").run(move.operationId);
+        expect(await service.runRebalancing()).toBe(false);
+        const failed = service.getRebalanceRun(run.id)!;
+        expect(failed.state).toBe('failed');
+        expect(failed.moves.find(item => item.id === move.id)).toMatchObject({ state: 'failed', message: 'Recovery needed' });
+        expect(failed.moves.filter(item => item.state === 'skipped')).toHaveLength(run.moves.length - 1);
+    });
+
+    it('keeps the source running until a global startup slot is available', async () => {
+        const run = await start();
+        const service = await import('../../src/lib/server/rebalancing/index.js');
+        await service.runRebalancing();
+        const move = await currentMove(run.id);
+        const { db } = await import('../../src/lib/server/db/index.js');
+        const { beginDeployment } = await import('../../src/lib/server/provisioning/history.js');
+        const occupied = beginDeployment('another-server', 'create', 'fixture');
+        await provision();
+        const { hostMigration } = await import('../../src/lib/server/provisioning/host-drain.js');
+        expect(hostMigration(move.operationId!)?.remove_operation_id).toBeNull();
+        db().prepare("UPDATE server_deployments SET state='done' WHERE id=?").run(occupied);
+        await provision();
+        expect(hostMigration(move.operationId!)?.remove_operation_id).toEqual(expect.any(String));
+    });
+
+    it('retains its global startup reservation between source removal and destination creation', async () => {
+        const run = await start();
+        const service = await import('../../src/lib/server/rebalancing/index.js');
+        await service.runRebalancing(); await provision();
+        const move = await currentMove(run.id);
+        const { hostMigration } = await import('../../src/lib/server/provisioning/migrations.js');
+        const { reportHostOperation } = await import('../../src/lib/server/hosts/index.js');
+        reportHostOperation(sourceId, hostMigration(move.operationId!)!.remove_operation_id!, { state: 'completed' });
+        await provision();
+        const { db } = await import('../../src/lib/server/db/index.js');
+        expect(db().prepare('SELECT state FROM server_deployments WHERE id=?').get(move.operationId)).toEqual({ state: 'waiting' });
+        const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
+        const extra = queueNativeCreation({ game: 't5', gameMode: 'zombie', mapSlug: 'theater',
+            hostId: targetId, libraryId, image, plutoKey: 'another-key' });
+        expect(extra.ok).toBe(true);
+        if (!extra.ok) throw new Error(extra.message);
+        await provision();
+        expect(db().prepare('SELECT operation_id FROM native_provisioning WHERE deployment_id=?').get(extra.deploymentId)).toBeUndefined();
+        expect(db().prepare('SELECT operation_id FROM native_provisioning WHERE deployment_id=?').get(move.operationId))
+            .toEqual({ operation_id: expect.any(String) });
+    });
+
+    it.each([{ state: 'running', healthy: false }, { state: 'stopped', healthy: false }])('holds removal when fresh inspection returns $state with healthy=$healthy', async health => {
+        const run = await start();
+        const service = await import('../../src/lib/server/rebalancing/index.js');
+        await service.runRebalancing();
+        const move = await currentMove(run.id);
+        const { nativeRequest } = await import('../../src/lib/server/backends/native.js');
+        vi.mocked(nativeRequest).mockResolvedValueOnce(health);
+        await provision();
+        const { hostMigration } = await import('../../src/lib/server/provisioning/host-drain.js');
+        expect(hostMigration(move.operationId!)?.remove_operation_id).toBeNull();
+        const { db } = await import('../../src/lib/server/db/index.js');
+        expect(db().prepare('SELECT stage FROM server_deployments WHERE id=?').get(move.operationId))
+            .toEqual({ stage: expect.stringContaining('no longer healthy') });
+    });
+});

tests/unit/rebalancing-snapshot.test.ts

@@ -0,0 +1,153 @@
+import { afterEach, beforeEach, expect, it, vi } from 'vitest';
+import { newFixture, type Fixture } from '../helpers/fixture.js';
+import { prepareNativeHost } from '../helpers/native-host.js';
+import type { HostInventory } from '../../src/lib/hosts.js';
+import type { RebalanceSettings } from '../../src/lib/rebalancing.js';
+
+// Artifact preservation has its own migration tests; these exercise live admission evidence.
+vi.mock('../../src/lib/server/provisioning/move-recipes.js', () => ({ captureMoveRecipe: vi.fn(() => ({})) }));
+const server = 't6_zombie_town_1', gib = 1024 ** 3;
+let fixture: Fixture;
+beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
+afterEach(async () => { vi.useRealTimers(); vi.restoreAllMocks(); await fixture.cleanup(); });
+
+async function setup() {
+    const source = await prepareNativeHost([server]), target = await prepareNativeHost();
+    const { db } = await import('../../src/lib/server/db/index.js');
+    const { ensureRebalancingSchema } = await import('../../src/lib/server/rebalancing/schema.js');
+    ensureRebalancingSchema(db());
+    const base = Date.now(); vi.useFakeTimers({ toFake: ['Date'] }); vi.setSystemTime(base);
+    const at = new Date().toISOString();
+    const native = db().prepare('SELECT id,generation,library_id FROM native_instances WHERE server=?').get(server) as { id: string; generation: number; library_id: string };
+    const original = db().prepare('SELECT inventory FROM ec_hosts WHERE id=?').get(source.host.id) as { inventory: string };
+    const inventory = JSON.parse(original.inventory) as HostInventory;
+    const resources = { instanceId: native.id, generation: native.generation, cpuCores: 2, memoryBytes: 2 * gib, uptimeSeconds: 600 };
+    db().prepare('UPDATE ec_hosts SET inventory=?,last_heartbeat=? WHERE id=?').run(JSON.stringify({ ...inventory, cpuCount: 8, cpuUsagePercent: 75,
+        memoryTotal: 16 * gib, memoryAvailable: 8 * gib, instanceResources: [resources] }), at, source.host.id);
+    db().prepare('UPDATE ec_hosts SET inventory=?,last_heartbeat=? WHERE id=?').run(JSON.stringify({ ...inventory, cpuCount: 8, cpuUsagePercent: 0,
+        memoryTotal: 16 * gib, memoryAvailable: 15 * gib, instanceResources: [] }), at, target.host.id);
+    // Target has a verified copy of the source's deployed library version.
+    const cache = db().prepare('SELECT manifest_sha256 FROM game_file_versions WHERE id=?').get(native.library_id) as { manifest_sha256: string };
+    const libraries = await import('../../src/lib/server/game-files/index.js');
+    libraries.updateHostLibraryCaches(target.host.id, [{ libraryVersionId: native.library_id, status: 'ready', manifestSha256: cache.manifest_sha256 }]);
+    db().prepare(`INSERT INTO servers(name,game,map,map_slug,container_state,container_checked_at,last_seen_at,last_checked_at,last_players)
+        VALUES(?,'t6','zm_transit','town','running',?,?,?,0)`).run(server, at, at, at);
+    db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES('key','t6','private-key')").run();
+    db().prepare("INSERT INTO server_key_leases(server_id,key_id) VALUES(?,'key')").run(server);
+    const settings: RebalanceSettings = { hostIds: [source.host.id, target.host.id], game: 'all', maxMoves: 10 };
+    const api = await import('../../src/lib/server/rebalancing/snapshot.js');
+    const planner = await import('../../src/lib/server/rebalancing/planner.js');
+    const read = (except = '') => api.rebalanceSnapshot(settings, except);
+    return { db, source, target, native, inventory, resources, settings, read, ...api, ...planner, base, at };
+}
+
+it('selects healthy empty native instances on active hosts without creating host work', async () => {
+    const s = await setup();
+    const before = s.db().prepare('SELECT COUNT(*) AS n FROM ec_host_operations').get();
+    const snapshot = s.read();
+    expect(snapshot.instances).toHaveLength(1);
+    const plan = s.planRebalance(snapshot, s.settings);
+    expect(plan.moves).toHaveLength(1);
+    expect(plan.moves[0]).toMatchObject({ sourceHostId: s.source.host.id, targetHostId: s.target.host.id });
+    expect(s.db().prepare('SELECT COUNT(*) AS n FROM ec_host_operations').get()).toEqual(before);
+    expect(s.rebalanceHosts().every(host => host.eligible)).toBe(true);
+    expect(s.read().hosts.find(host => host.id === s.source.host.id)?.used.cpuCores).toBe(6);
+});
+
+it.each(['occupied', 'unknown', 'stopped', 'maintenance', 'recovery', 'lifecycle', 'generation', 'stale-host', 'future-host', 'startup', 'artifacts'] as const)
+    ('excludes %s instances with an explanation', async condition => {
+        const s = await setup();
+        if (condition === 'occupied') s.db().prepare('UPDATE servers SET last_players=1').run();
+        if (condition === 'unknown') s.db().prepare('UPDATE servers SET last_players=NULL').run();
+        if (condition === 'stopped') s.db().prepare("UPDATE servers SET desired_state='stopped'").run();
+        if (condition === 'maintenance') s.db().prepare('INSERT INTO maintenance_jobs(server,requested_at) VALUES(?,?)').run(server, s.at);
+        if (condition === 'recovery') s.db().prepare('INSERT INTO restart_rollout(server,requested_at) VALUES(?,?)').run(server, s.at);
+        if (condition === 'lifecycle') s.db().prepare(`INSERT INTO ec_host_operations(id,host_id,instance_id,generation,kind,payload,state,created_at,updated_at)
+            VALUES('pending',?,?,1,'instance.stop','{}','unknown',?,?)`).run(s.source.host.id, s.native.id, s.at, s.at);
+        if (condition === 'generation') s.db().prepare('UPDATE native_instances SET generation=2').run();
+        if (condition === 'stale-host' || condition === 'future-host') s.db().prepare('UPDATE ec_hosts SET last_heartbeat=? WHERE id=?')
+            .run(new Date(s.base + (condition === 'stale-host' ? -91_000 : 1)).toISOString(), s.source.host.id);
+        if (condition === 'startup') s.db().prepare("UPDATE ec_hosts SET inventory=json_set(inventory,'$.instanceResources[0].uptimeSeconds',200) WHERE id=?").run(s.source.host.id);
+        if (condition === 'artifacts') vi.mocked((await import('../../src/lib/server/provisioning/move-recipes.js')).captureMoveRecipe).mockImplementationOnce(() => { throw new Error('Deployed files are unavailable.'); });
+        const snapshot = s.read();
+        expect(snapshot.instances).toEqual([]);
+        expect(snapshot.exclusions[0]).toMatchObject({ server, reason: expect.any(String) });
+    });
+
+it('honors a current positive RCON veto even when telemetry reports empty', async () => {
+    const s = await setup(), status = await import('../../src/lib/server/fleet/status.js');
+    const observe = status.observationReader();
+    vi.spyOn(status, 'observationReader').mockReturnValue(row => ({ ...observe(row), last_players: 0, status_source: 'telemetry' }));
+    s.db().prepare('UPDATE servers SET last_players=1').run();
+    expect(s.read().exclusions[0].reason).toContain('RCON');
+});
+
+it('uses conservative recent host pressure and credits only matching observed generation relief', async () => {
+    const s = await setup();
+    s.db().prepare(`INSERT INTO ec_host_resource_samples(host_id,sampled_at,cpu_count,cpu_usage_percent,memory_total,memory_available)
+        VALUES(?,?,8,90,?,?)`).run(s.source.host.id, new Date(s.base - 60_000).toISOString(), 16 * gib, 6 * gib);
+    s.db().prepare(`INSERT INTO ec_instance_resource_samples(host_id,instance_id,generation,phase,bucket,cpu_cores,memory_bytes)
+        VALUES(?,?,1,'running',?,1,?)`).run(s.source.host.id, s.native.id, new Date(s.base - 60_000).toISOString(), gib);
+    const snapshot = s.read();
+    expect(snapshot.hosts.find(host => host.id === s.source.host.id)?.used).toEqual({ cpuCores: 7.2, memoryBytes: 10 * gib });
+    expect(snapshot.instances[0].relief).toEqual({ cpuCores: 1, memoryBytes: gib });
+    expect(snapshot.instances[0].demand).toEqual({ cpuCores: 2, memoryBytes: 2 * gib });
+});
+
+it('counts a pending migration once before and after ownership changes and never assumes unplaced pinning', async () => {
+    const s = await setup();
+    s.db().prepare(`INSERT INTO server_deployments(id,server,action,state,stage,started_at,triggered_by)
+        VALUES('moving',?,'rebuild','waiting','Move',?,'test')`).run(server, s.at);
+    s.db().prepare('INSERT INTO native_provisioning(deployment_id,instance_id,generation) VALUES(?,?,1)').run('moving', s.native.id);
+    s.db().prepare(`INSERT INTO ec_host_migrations(deployment_id,source_host_id,instance_id,target_host_id)
+        VALUES('moving',?,?,?)`).run(s.source.host.id, s.native.id, s.target.host.id);
+    const target = () => s.read().hosts.find(host => host.id === s.target.host.id)!;
+    expect(target().used.cpuCores).toBe(0.5);
+    expect(target().used.memoryBytes).toBe(2 * gib);
+    expect(s.read('moving').hosts.find(host => host.id === s.target.host.id)?.used.cpuCores).toBe(0);
+    s.db().prepare('UPDATE native_instances SET host_id=?,generation=2').run(s.target.host.id);
+    s.db().prepare('UPDATE native_provisioning SET generation=2').run();
+    s.db().prepare('UPDATE ec_host_migrations SET removed_at=?').run(s.at);
+    expect(target().used.cpuCores).toBe(0.5);
+    expect(target().used.memoryBytes).toBe(2 * gib);
+    s.db().prepare("UPDATE ec_hosts SET inventory=json_set(inventory,'$.instanceResources',json(?),'$.cpuUsagePercent',25,'$.memoryAvailable',?) WHERE id=?")
+        .run(JSON.stringify([{ ...s.resources, generation: 2 }]), 13 * gib, s.target.host.id);
+    expect(target().used).toEqual({ cpuCores: 2, memoryBytes: 3 * gib });
+    s.db().prepare(`INSERT INTO server_deployments(id,server,action,state,stage,started_at,triggered_by)
+        VALUES('unplaced','t6_zombie_town_2','create','queued','Place',?,'test')`).run(s.at);
+    s.db().prepare('INSERT INTO native_creation_intents(deployment_id,selection) VALUES(?,?)').run('unplaced', JSON.stringify({ hostId: s.target.host.id }));
+    expect(target().used).toEqual({ cpuCores: 2, memoryBytes: 3 * gib });
+    expect(target().instances).toBe(1);
+});
+
+it('allows admission to ignore its own unsubmitted rebuild while retaining the source empty evidence', async () => {
+    const s = await setup();
+    s.db().prepare(`INSERT INTO server_deployments(id,server,action,state,stage,started_at,triggered_by)
+        VALUES('own',?,'rebuild','queued','Move',?,'rebalance:test')`).run(server, s.at);
+    expect(s.read().instances).toHaveLength(0);
+    expect(s.read('own').instances).toHaveLength(1);
+});
+
+it('enforces the 30 minute cooldown across generations after a successful move', async () => {
+    const s = await setup();
+    s.db().prepare(`INSERT INTO rebalancing_runs(id,settings,state,created_at,updated_at,message) VALUES('old','{}','done',?,?,'Done')`).run(s.at, s.at);
+    s.db().prepare(`INSERT INTO rebalancing_moves(id,run_id,ordinal,server,instance_id,generation,source_host_id,target_host_id,candidate,state,message,updated_at)
+        VALUES('m','old',0,?,?,0,?,?,'{}','done','Done',?)`).run(server, s.native.id, s.target.host.id, s.source.host.id, s.at);
+    expect(s.read().exclusions[0].reason).toContain('30 minutes');
+    s.db().prepare('UPDATE rebalancing_moves SET updated_at=?').run(new Date(s.base - 30 * 60_000 - 1).toISOString());
+    expect(s.read().instances).toHaveLength(1);
+});
+
+it('credits a freed game port only when the current range and bind will actually gain a port', async () => {
+    const s = await setup();
+    expect(s.read().instances[0].releasesPort).toBe(false);
+    s.db().prepare(`INSERT INTO ec_host_ports(host_id,instance_id,bind_address,protocol,port,created_at)
+        VALUES(?,?,'0.0.0.0','udp',28960,?)`).run(s.source.host.id, s.native.id, s.at);
+    expect(s.read().instances[0].releasesPort).toBe(true);
+    s.db().prepare('UPDATE ec_hosts SET port_min=28970 WHERE id=?').run(s.source.host.id);
+    expect(s.read().instances[0].releasesPort).toBe(false);
+    s.db().prepare('UPDATE ec_hosts SET port_min=28960 WHERE id=?').run(s.source.host.id);
+    s.db().prepare(`INSERT INTO ec_host_ports(host_id,instance_id,bind_address,protocol,port,created_at)
+        VALUES(?,'other','127.0.0.2','tcp',28960,?)`).run(s.source.host.id, s.at);
+    expect(s.read().instances[0].releasesPort).toBe(false);
+});

tests/unit/retention-storage.test.ts

@@ -45,6 +45,34 @@ it('expires completed file payloads while keeping history, replay identity, hold
     expect(pruneHistory('operation-payload')).toBe(0);
 });
 
+it('retains rebalance recovery files until the move is reconciled, then lets history expire', async () => {
+    const { db } = await import('../../src/lib/server/db/index.js');
+    const { ensureHostSchema } = await import('../../src/lib/server/hosts/schema.js');
+    const { pruneHistory } = await import('../../src/lib/server/retention.js');
+    const { putFileBlob } = await import('../../src/lib/server/db/file-blobs.js');
+    const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
+    ensureHostSchema(db());
+    saveFleetSettings({ ...fleetSettings(), historyKeepCount: 0, provisioningKeepDays: 1, payloadKeepDays: 1 });
+    db().prepare(`INSERT INTO server_deployments(id,server,action,state,stage,started_at,finished_at,triggered_by)
+        VALUES('move','s','rebuild','done','healthy',?,?,'rebalance:run')`).run(old, old);
+    const hash = putFileBlob(Buffer.from('frozen deployed file'));
+    db().prepare('INSERT INTO server_launch_recipes(deployment_id,recipe) VALUES(?,?)')
+        .run('move', JSON.stringify({ files: [{ path: 'script.gsc', sha256: hash, contentHash: hash }] }));
+    db().prepare(`INSERT INTO rebalancing_runs(id,settings,state,created_at,updated_at,message)
+        VALUES('run','{}','running',?,?,'waiting')`).run(old, old);
+    db().prepare(`INSERT INTO rebalancing_moves(id,run_id,ordinal,server,instance_id,generation,source_host_id,target_host_id,candidate,state,operation_id,message,updated_at)
+        VALUES('m','run',0,'s','i',1,'a','b','{}','moving','move','waiting',?)`).run(old);
+    expect(pruneHistory('workflow-payload')).toBe(0);
+    expect(pruneHistory('provisioning')).toBe(0);
+    db().prepare("UPDATE rebalancing_moves SET state='failed'").run();
+    expect(pruneHistory('workflow-payload')).toBe(0);
+    expect(pruneHistory('provisioning')).toBe(0);
+    db().prepare("UPDATE rebalancing_moves SET state='done'").run();
+    expect(pruneHistory('workflow-payload')).toBe(1);
+    expect(pruneHistory('provisioning')).toBe(1);
+    expect(db().prepare('SELECT state,operation_id FROM rebalancing_moves').get()).toEqual({ state: 'done', operation_id: null });
+});
+
 it('normalizes legacy retention clocks once and reuses clock indexes', () => {
     const handle = new Database(':memory:');
     try {

tests/unit/ui-rebalancing.test.ts

@@ -0,0 +1,56 @@
+import { expect, it } from 'vitest';
+import { render } from 'svelte/server';
+import type { RebalanceCandidate, RebalancePreview, RebalanceRun } from '../../src/lib/rebalancing.js';
+import RebalancePreviewView from '../../src/lib/ui/components/capacity/RebalancePreviewView.svelte';
+import RebalanceRunView from '../../src/lib/ui/components/capacity/RebalanceRunView.svelte';
+
+const at = '2026-10-08T12:00:00Z';
+const move: RebalanceCandidate = { server: 't5_zombie_theater_1', instanceId: 'instance-a', generation: 1,
+    sourceHostId: 'source-a', sourceHostName: 'Source A', targetHostId: 'target-b', targetHostName: 'Target B', improvement: 0.1 };
+const preview: RebalancePreview = { id: 'preview-a', createdAt: at, expiresAt: '2026-10-08T12:05:00Z',
+    settings: { hostIds: ['source-a', 'target-b'], game: 'all', maxMoves: 5 }, moves: [move],
+    hosts: [{ id: 'source-a', name: 'Source A', before: { cpuPercent: 75, memoryPercent: 50, instances: 6 }, after: { cpuPercent: 62.5, memoryPercent: 42, instances: 5 } }],
+    exclusions: [{ server: 't6_zm_transit_1', reason: 'Lobby has players.' }], warnings: ['Occupied lobbies stay in place.'] };
+const run: RebalanceRun = { id: 'run-a', createdAt: at, updatedAt: at, settings: preview.settings, state: 'running', message: 'Moving one lobby.',
+    moves: [{ ...move, id: 'move-a', state: 'moving', operationId: 'operation-a', message: 'Waiting for destination health.', updatedAt: at }] };
+const noop = () => {};
+
+it('shows current and projected resources, the move destination, and exclusion reasons before starting', () => {
+    const body = render(RebalancePreviewView, { props: { preview, expired: false, onstart: noop } }).body;
+    expect(body).toContain('6 → 5');
+    expect(body).toContain('75% → 62.5%');
+    expect(body).toContain('50% → 42%');
+    expect(body).toContain('Source A → Target B');
+    expect(body).toContain('Lobby has players.');
+    expect(body).toContain('Occupied lobbies stay in place.');
+    expect(body).toContain('aria-label="Projected host distribution" tabindex="0"');
+    expect(body).not.toMatch(/<button[^>]*disabled/);
+});
+
+it.each(['expired', 'empty', 'blocked'] as const)('prevents starting a %s preview', condition => {
+    const body = render(RebalancePreviewView, { props: {
+        preview: condition === 'empty' ? { ...preview, moves: [] } : preview,
+        expired: condition === 'expired', disabled: condition === 'blocked', onstart: noop
+    } }).body;
+    expect(body).toMatch(/<button[^>]*disabled/);
+    if (condition === 'expired') expect(body).toContain('This preview has expired.');
+    if (condition === 'empty') expect(body).toContain('No eligible moves would improve');
+});
+
+it('keeps run progress tied to its provisioning record and explains stopping semantics', () => {
+    const body = render(RebalanceRunView, { props: { run, canManage: true, onstop: noop } }).body;
+    expect(body).toContain('0 of 1 moves completed');
+    expect(body).toContain('Waiting for destination health.');
+    expect(body).toContain('href="/provisioning/operation-a?returnTo=%2Fcapacity%3Frebalance%3D1%26rebalanceRun%3Drun-a"');
+    expect(body).toContain('Stop remaining moves');
+    expect(body).toContain('A move already in progress continues through health verification.');
+});
+
+it('hides stop from viewers and terminal or already-stopping runs', () => {
+    expect(render(RebalanceRunView, { props: { run, canManage: false, onstop: noop } }).body).not.toContain('Stop remaining moves');
+    for (const state of ['stopping', 'done', 'stopped', 'failed'] as const) {
+        const body = render(RebalanceRunView, { props: { run: { ...run, state }, canManage: true, onstop: noop } }).body;
+        expect(body).not.toContain('Stop remaining moves');
+        expect(body).toContain('View provisioning');
+    }
+});