diff --git a/docs/autoscaling.md b/docs/autoscaling.md index 2e80e31..ad8df39 100644 --- a/docs/autoscaling.md +++ b/docs/autoscaling.md @@ -1,12 +1,12 @@ # Lobby autoscaling -Open **Fleet → Autoscaling** to keep empty lobbies available for new groups. Policies apply to a specific game and map variant. A partly occupied Zombies lobby is not an empty spare, even when it still has player slots. +Open **Fleet → Autoscaling** to keep empty lobbies available for new groups. Policies apply to one map variant or every map in a game. A game-wide policy applies the same minimum, maximum, and spare targets independently to each map, including all variants. A partly occupied Zombies lobby is not an empty spare, even when it still has player slots. Autoscaling is opt-in. Installing the feature creates no policies and changes no lobby counts. Use a two-week dry run before enabling live actions. ## Configure a policy -Choose **Add policy**, select the game and map, and enter an explicit maximum. The initial editor selects Dry run, with a minimum of one running lobby, one empty spare, a 20-minute empty interval, and a five-minute action cooldown. Counts must be between 1 and 20; the minimum and spare target cannot exceed the maximum. +Choose **Add policy**, select the game and scope, and enter an explicit maximum. The initial editor selects Dry run, with a minimum of one running lobby, one empty spare, a 20-minute empty interval, and a five-minute action cooldown. Counts must be between 1 and 20; the minimum and spare target cannot exceed the maximum. Choose **Preview changes** to inspect the existing lobbies, saved key coverage, target, proposed action, and blockers. **Save reviewed policy** checks the policy revision and current fleet again. If the preview changed, review a new preview before saving. @@ -19,11 +19,11 @@ Choose **Preview changes** to inspect the existing lobbies, saved key coverage, Live activation requires saved Plutonium keys for enough exact lobby identities to support the configured maximum. Keys may be nonconsecutive: if only lobby 3 and lobby 8 are keyed, those identities can be used. Existing lobbies and unfinished operations must be healthy and settled before activation. Automatic removal also requires agent **1.7.0 or later** advertising `instance.autoscaling-remove` on each affected host. -The desired count is `min(maximum, max(minimum, occupied lobbies + empty spare target))`. The controller evaluates policies every 15 seconds and requests at most one unresolved operation per map. Create uses the existing startup concurrency, CPU, memory, library, port, and host admission checks. Lack of host capacity leaves one request waiting; it does not create additional requests every tick. Creates use current promoted launch files when provisioning begins. +The desired count is `min(maximum, max(minimum, occupied lobbies + empty spare target))`. The controller evaluates policies every 15 seconds and requests at most one unresolved operation per policy. Game-wide policies share one cooldown and one outstanding operation across their maps, prioritizing additions before removals. Create uses the existing startup concurrency, CPU, memory, library, port, and host admission checks. Lack of host capacity leaves one request waiting; it does not create additional requests every tick. Creates use current promoted launch files when provisioning begins. ## Ownership and blockers -A live policy owns the map's lobby count. Competing collection launches and manual Create, Start, Stop, Rebuild, or Destroy requests are rejected with a policy link. Exact collection matching also checks managed maps absent from the collection. To hand control back, set the policy to Off and wait for unsettled operations to finish. +A live policy owns the lobby counts for its selected maps. Overlapping live game-wide and map-specific policies are rejected; dry-run policies may overlap. Game-wide rules include every catalog map and require key coverage for the maximum on each map. Competing collection launches and manual Create, Start, Stop, Rebuild, or Destroy requests are rejected with a policy link. Exact collection matching also checks managed maps absent from the collection. To hand control back, set the policy to Off and wait for unsettled operations to finish. Unknown or stale occupancy, stopped or unhealthy members, active betas, campaigns, maintenance, host drains, and unresolved recovery hold the policy's decisions. Autoscaling does not repair these conditions itself. The page and linked Provisioning operation explain the current blocker, including capacity and missing-key waits. @@ -31,7 +31,7 @@ Current positive occupancy from either telemetry or RCON vetoes an empty claim. ## Pause and cancellation -Use **Pause all** to stop new decisions globally, or **Edit or pause → Off** for one map. A global pause retains each live policy's ownership; setting it Off releases ownership once its outstanding operation settles. Resuming the global switch preserves each policy's selected mode. +Use **Pause all** to stop new decisions globally, or **Edit or pause → Off** for one policy. A global pause retains each live policy's ownership; setting it Off releases ownership once its outstanding operation settles. Resuming the global switch preserves each policy's selected mode. Unsubmitted creates and queued removals are withdrawn where possible. Cancelling an autoscaling operation from Provisioning also turns its policy Off so it cannot immediately request a replacement. @@ -59,6 +59,6 @@ Measurement totals exclude collection gaps. Request-to-verified-readiness summar `GET /api/autoscaling` returns policies, current evidence, recent decisions, measurements, and global pause state. Viewers may read it; mutations require an operator or administrator, using the existing authentication and origin checks. -`POST /api/autoscaling` accepts `preview`, `save`, or `global-pause`. Preview/save requests contain `settings`, plus `id` and `revision` when editing. Save also requires the preview `token`. Global pause uses a boolean `paused` field. Agent removal claims use a separate host-authenticated endpoint and are bound to the exact command, host, instance generation, policy revision, and expiry. +`POST /api/autoscaling` accepts `preview`, `save`, or `global-pause`. Preview/save requests contain `settings`, plus `id` and `revision` when editing. Use `mapSlug: '*'` for every map in the selected game; existing map-specific requests are unchanged. Scope cannot change when editing. Save also requires the preview `token`. Global pause uses a boolean `paused` field. Agent removal claims use a separate host-authenticated endpoint and are bound to the exact command, host, instance generation, policy revision, and expiry. Shared contracts live in `src/lib/autoscaling.ts`; the evaluator, snapshots, persistence, ownership, and authorization live in `src/lib/server/autoscaling/`. Provisioning remains the only lifecycle executor. See the [approved implementation plan](lobby-autoscaling-plan.md) for the design and rollout criteria. diff --git a/docs/lobby-autoscaling-plan.md b/docs/lobby-autoscaling-plan.md index da24bba..3bf607c 100644 --- a/docs/lobby-autoscaling-plan.md +++ b/docs/lobby-autoscaling-plan.md @@ -146,3 +146,5 @@ Validation includes planner and snapshot tests, duplicate-admission and stale-pr The full controller suite reported 1,934 passing tests and two existing failures in `server-detail-api.test.ts` (plugin classification) and `watchdog.test.ts` (an occupied-telemetry RCON expectation). Both failures reproduce independently at the pre-implementation commit `a22461a`. They are not autoscaling regressions. Targeted controller tests, all 228 Python agent tests, type checking, and the production build passed. An isolated Firefox review exercised policy creation, preview, save, keyboard focus, and 1440, 390, and 320 pixel layouts. A narrow-layout heading issue found during review was corrected. The preview used a temporary database and no production host credentials. Production CPU savings, external joinability, spare targets, and acceptable deficit durations still require the two-week observation period and operator-run pilot. + +Game-wide policy follow-up: selecting every map applies the targets independently to each catalog map and variant. Game-wide policies serialize actions across maps and cannot overlap live map-specific ownership. Existing map policies remain compatible. diff --git a/src/lib/autoscaling.ts b/src/lib/autoscaling.ts index 5543815..0b4947d 100644 --- a/src/lib/autoscaling.ts +++ b/src/lib/autoscaling.ts @@ -3,14 +3,17 @@ import { zombieMapBySlug, type ZombieGame } from './map-aliases.js'; export const autoscalingModes = ['off', 'dry-run', 'create-only', 'create-destroy'] as const; export type AutoscalingMode = typeof autoscalingModes[number]; export interface ScalingSettings { - game: ZombieGame; mapSlug: string; mode: AutoscalingMode; + game: ZombieGame; + /** '*' applies the targets independently to every map. */ + mapSlug: string; + mode: AutoscalingMode; minimum: number; maximum: number; spares: number; emptyMinutes: number; cooldownMinutes: number; } export interface ScalingPolicy extends ScalingSettings { id: string; revision: number; nextActionAt: number; pausedReason: string | null; updatedAt: string; } export interface ScalingMember { - name: string; identity: string; state: 'occupied' | 'empty' | 'creating' | 'removing' | 'stopped' | 'unknown' | 'protected'; + name: string; mapSlug?: string; identity: string; state: 'occupied' | 'empty' | 'creating' | 'removing' | 'stopped' | 'unknown' | 'protected'; players: number | null; observedAt: number | null; source: string | null; emptySince: number | null; blocker: string | null; operationId: string | null; emptyStartedAt?: number | null; @@ -20,7 +23,7 @@ export interface ScalingSnapshot { } export interface ScalingDecision { action: 'create' | 'destroy' | 'wait'; reason: string; server: string | null; - desired: number; occupied: number; empty: number; pending: number; unknown: number; + desired: number; deficit: number; occupied: number; empty: number; pending: number; unknown: number; } export interface ScalingPreview { policy: ScalingPolicy; snapshot: ScalingSnapshot; decision: ScalingDecision; token: string; blockers: string[] } export interface ScalingHistory { id: string; at: number; lastSeenAt: number; decision: ScalingDecision; operationId: string | null; operationState: string | null } @@ -38,7 +41,7 @@ export function scalingSettings(input: unknown): ScalingSettings { if (!input || typeof input !== 'object' || Array.isArray(input)) throw new Error('Provide an autoscaling policy.'); const value = input as Record; const { game, mapSlug } = value; - if ((game !== 't5' && game !== 't6') || typeof mapSlug !== 'string' || zombieMapBySlug(game, mapSlug)?.slug !== mapSlug) throw new Error('Choose a supported game and map variant.'); + if ((game !== 't5' && game !== 't6') || typeof mapSlug !== 'string' || (mapSlug !== '*' && zombieMapBySlug(game, mapSlug)?.slug !== mapSlug)) throw new Error('Choose a supported game and map variant, or all maps.'); const mode = value.mode ?? 'off'; if (!autoscalingModes.includes(mode as AutoscalingMode)) throw new Error('Choose a valid autoscaling mode.'); const integer = (key: string, fallback: number | undefined, max: number) => { diff --git a/src/lib/server/autoscaling/index.ts b/src/lib/server/autoscaling/index.ts index ad8ea78..6ec369a 100644 --- a/src/lib/server/autoscaling/index.ts +++ b/src/lib/server/autoscaling/index.ts @@ -2,6 +2,7 @@ import { createHash, randomUUID } from 'node:crypto'; import { db } from '../db/index.js'; import { liveScaling, scalingSettings, type ScalingPolicy, type ScalingPreview, type ScalingReport, type ScalingView } from '../../autoscaling.js'; import { emptyObservation, planScaling } from './planner.js'; +import { zombieMaps, parseServerName } from '../../map-aliases.js'; import { scalingSnapshot } from './snapshot.js'; import { nativeLaunchSelection, queueNativeCreationIntent } from '../provisioning/native.js'; import { cancelCreation } from '../provisioning/cancel-creation.js'; @@ -10,6 +11,7 @@ import { destroyServer } from '../provisioning/instances.js'; import { cancelRemoval } from '../provisioning/history.js'; import { autoscalingRemovalCapability, revokePolicyRemovals } from './authorization.js'; import { getHost } from '../hosts/index.js'; +import { scalingOwner } from './ownership.js'; import { getServer } from '../fleet/servers.js'; export class ScalingError extends Error { constructor(message: string, public status = 400) { super(message); } } @@ -29,7 +31,11 @@ export function previewScaling(input: unknown, id?: string, revision?: number): const snapshot = scalingSnapshot(policy); const blockers: string[] = []; if (liveScaling(policy.mode)) { - if (snapshot.keyedNames.length < policy.maximum) blockers.push(`Save keys for at least ${policy.maximum} lobby identities before activating this policy.`); + const owner = scalingOwner(policy.game, policy.mapSlug); + if (owner && owner !== policy.id) blockers.push(`An overlapping policy owns this lineup. Pause it and settle pending work first: /autoscaling#${owner}`); + for (const map of zombieMaps(policy.game).filter(m => policy.mapSlug === '*' || m.slug === policy.mapSlug)) { + if (snapshot.keyedNames.filter(name => parseServerName(name)?.mapSlug === map.slug).length < policy.maximum) blockers.push(`Save keys for at least ${policy.maximum} lobby identities for ${map.friendly} before activating this policy.`); + } blockers.push(...snapshot.blockers); for (const member of snapshot.members) if (!['occupied', 'empty'].includes(member.state)) blockers.push(`${member.name}: ${member.blocker ?? member.state}. Settle this work before activation.`); if (policy.mode === 'create-destroy') for (const member of snapshot.members) { @@ -52,7 +58,7 @@ export function saveScaling(input: unknown, id?: string, revision?: number, toke if (preview.blockers.length) throw new ScalingError(preview.blockers[0]); const policy = preview.policy, settings = scalingSettings(policy), nextId = id ?? randomUUID(), at = new Date().toISOString(); if (id) revokePolicyRemovals(id); - if (!id && db().prepare('SELECT 1 FROM autoscaling_policies WHERE game=? AND map_slug=?').get(policy.game, policy.mapSlug)) throw new ScalingError('A policy already exists for this map.', 409); + if (!id && db().prepare('SELECT 1 FROM autoscaling_policies WHERE game=? AND map_slug=?').get(policy.game, policy.mapSlug)) throw new ScalingError('A policy already exists for this scope.', 409); db().prepare(`INSERT INTO autoscaling_policies(id,game,map_slug,settings,mode,updated_at) VALUES(?,?,?,?,?,?) ON CONFLICT(id) DO UPDATE SET settings=excluded.settings,mode=excluded.mode,revision=autoscaling_policies.revision+1,paused_reason=NULL,updated_at=excluded.updated_at`).run(nextId, policy.game, policy.mapSlug, JSON.stringify(settings), policy.mode, at); // An edit starts a new observation interval; earlier inactivity is not authorization. @@ -139,7 +145,7 @@ export async function runAutoscaling(): Promise { try { db().transaction(() => { const result = decision.action === 'create' - ? queueNativeCreationIntent({ server: decision.server!, game: policy.game, mapSlug: policy.mapSlug, + ? queueNativeCreationIntent({ server: decision.server!, game: policy.game, mapSlug: parseServerName(decision.server!)!.mapSlug, selection: nativeLaunchSelection({ automatic: true }, policy.game), createdBy: `autoscaling:${policy.id}`, autoscalingDecisionId: decisionId }) : destroyServer(decision.server!, `autoscaling:${policy.id}`, decisionId); if (!result.ok) throw new Error(result.message); @@ -155,7 +161,7 @@ export async function runAutoscaling(): Promise { } } db().prepare(`INSERT OR IGNORE INTO autoscaling_samples(policy_id,sampled_at,mode,occupied,empty,unknown,deficit) VALUES(?,?,?,?,?,?,?)`) - .run(policy.id, Math.floor(now / 15_000) * 15_000, policy.mode, decision.occupied, decision.empty, decision.unknown, Math.max(0, policy.spares - decision.empty)); + .run(policy.id, Math.floor(now / 15_000) * 15_000, policy.mode, decision.occupied, decision.empty, decision.unknown, decision.deficit); })(); } return submitted; diff --git a/src/lib/server/autoscaling/ownership.ts b/src/lib/server/autoscaling/ownership.ts index 5fb658f..93f0ec7 100644 --- a/src/lib/server/autoscaling/ownership.ts +++ b/src/lib/server/autoscaling/ownership.ts @@ -3,7 +3,7 @@ import { getServer } from '../fleet/servers.js'; import { mapIdentity } from './snapshot.js'; export function scalingOwner(game: string, mapSlug: string): string | null { - return (db().prepare('SELECT id FROM autoscaling_policies WHERE game=? AND map_slug=? AND owns_map=1').get(game, mapSlug) as { id: string } | undefined)?.id ?? null; + return (db().prepare("SELECT id FROM autoscaling_policies WHERE game=? AND (map_slug=? OR map_slug='*' OR ?='*') AND owns_map=1").get(game, mapSlug, mapSlug) as { id: string } | undefined)?.id ?? null; } export function scalingOwnershipBlocker(server: string): string | null { const identity = mapIdentity(getServer(server) ?? { name: server }); diff --git a/src/lib/server/autoscaling/planner.ts b/src/lib/server/autoscaling/planner.ts index 0476192..2ecc876 100644 --- a/src/lib/server/autoscaling/planner.ts +++ b/src/lib/server/autoscaling/planner.ts @@ -1,4 +1,5 @@ import type { ScalingDecision, ScalingMember, ScalingPolicy, ScalingSnapshot } from '../../autoscaling.js'; +import { parseServerName, zombieMaps } from '../../map-aliases.js'; import { ONLINE_MAX_MS } from '../../server-freshness.js'; export interface EmptyObservation { identity: string; observedAt: number | null; emptySince: number | null; evaluatedAt: number } @@ -14,13 +15,32 @@ export function emptyObservation(member: ScalingMember, previous: EmptyObservati } export function planScaling(policy: ScalingPolicy, snapshot: ScalingSnapshot, now: number): ScalingDecision { + if (policy.mapSlug === '*') { + const maps = zombieMaps(policy.game); + const decisions = maps.map(map => { + const matches = (name: string) => (snapshot.members.find(m => m.name === name)?.mapSlug ?? parseServerName(name)?.mapSlug) === map.slug; + return planScaling({ ...policy, mapSlug: map.slug }, { + ...snapshot, members: snapshot.members.filter(m => matches(m.name)), + reservedNames: snapshot.reservedNames.filter(matches), keyedNames: snapshot.keyedNames.filter(matches) + }, now); + }); + // Serialize game-wide work and prefer replenishing any map before trimming another. + const pending = snapshot.members.find(m => m.state === 'creating' || m.state === 'removing'); + const selected = decisions.find(d => d.action === 'create') ?? decisions.find(d => d.action === 'destroy') + ?? decisions.find(d => d.deficit > 0 || d.unknown > 0) ?? decisions[0]; + const map = maps[decisions.indexOf(selected)]; + return { ...selected, action: pending ? 'wait' : selected.action, server: pending ? null : selected.server, + reason: pending ? pending.blocker ?? `Waiting for ${pending.name}: ${pending.state}.` : `${map.friendly}: ${selected.reason}`, + desired: decisions.reduce((n, d) => n + d.desired, 0), deficit: decisions.reduce((n, d) => n + d.deficit, 0), occupied: decisions.reduce((n, d) => n + d.occupied, 0), + empty: decisions.reduce((n, d) => n + d.empty, 0), pending: decisions.reduce((n, d) => n + d.pending, 0), unknown: decisions.reduce((n, d) => n + d.unknown, 0) }; + } const { members } = snapshot; const occupied = members.filter(m => m.state === 'occupied').length, empty = members.filter(m => m.state === 'empty').length; const pending = members.filter(m => m.state === 'creating' || m.state === 'removing').length; const unknown = members.filter(m => m.state === 'unknown' || m.state === 'stopped').length; const desired = Math.min(policy.maximum, Math.max(policy.minimum, occupied + policy.spares)); const result = (action: ScalingDecision['action'], reason: string, server: string | null = null): ScalingDecision => - ({ action, reason, server, desired, occupied, empty, pending, unknown }); + ({ action, reason, server, desired, deficit: Math.max(0, policy.spares - empty), occupied, empty, pending, unknown }); if (policy.mode === 'off') return result('wait', policy.pausedReason ?? 'Policy is off.'); if (policy.pausedReason) return result('wait', policy.pausedReason); if (snapshot.blockers.length) return result('wait', snapshot.blockers[0]); diff --git a/src/lib/server/autoscaling/snapshot.ts b/src/lib/server/autoscaling/snapshot.ts index c704387..c20b305 100644 --- a/src/lib/server/autoscaling/snapshot.ts +++ b/src/lib/server/autoscaling/snapshot.ts @@ -28,7 +28,7 @@ export function protectedWork(server: string, exceptOperation = ''): string | nu export function scalingSnapshot(policy: ScalingPolicy, now = Date.now(), exceptOperation = ''): ScalingSnapshot { const handle = db(), observe = observationReader(); const rows = listServers(); - const matches = (value: Parameters[0]) => { const identity = mapIdentity(value); return identity?.game === policy.game && identity.mapSlug === policy.mapSlug; }; + const matches = (value: Parameters[0]) => { const identity = mapIdentity(value); return identity?.game === policy.game && (policy.mapSlug === '*' || identity.mapSlug === policy.mapSlug); }; const blockers: string[] = []; for (const row of rows) if (!mapIdentity(row) && (row.game ?? parseServerName(row.name)?.game) === policy.game) blockers.push(`Resolve ambiguous map identity for ${row.name}.`); const operations = handle.prepare(`SELECT id,server,action,state,stage FROM server_deployments WHERE server_deleted_at IS NULL @@ -54,12 +54,12 @@ export function scalingSnapshot(policy: ScalingPolicy, now = Date.now(), exceptO else if (row.desired_state === 'stopped' || ['stopped','exited','created','dead'].includes(row.container_state ?? '')) state = 'stopped'; else if (observation.health === 'online' && players !== null && observedAt !== null && observedAt <= now && freshStatusTimestamp(new Date(observedAt).toISOString(), now)) state = players > 0 ? 'occupied' : 'empty'; const transitions = [row.inactive_since, observation.inactive_since].map(at => Date.parse(at ?? '')).filter(at => Number.isFinite(at) && at <= now); - return { name: row.name, identity, state, players, observedAt, source: source ?? null, emptySince: null, + return { name: row.name, mapSlug: mapIdentity(row)!.mapSlug, identity, state, players, observedAt, source: source ?? null, emptySince: null, emptyStartedAt: transitions.length ? Math.max(...transitions) : null, blocker, operationId: operation?.id ?? null }; }); for (const row of reservations.filter(matches)) if (!members.some(m => m.name === row.name)) { const operation = operations.find(o => o.server === row.name); - members.push({ name: row.name, identity: 'reserved', state: 'creating', players: null, observedAt: null, source: null, emptySince: null, blocker: operation?.stage ?? 'Creation is pending.', operationId: operation?.id ?? null }); + members.push({ name: row.name, mapSlug: row.map_slug, identity: 'reserved', state: 'creating', players: null, observedAt: null, source: null, emptySince: null, blocker: operation?.stage ?? 'Creation is pending.', operationId: operation?.id ?? null }); } const retained = handle.prepare('SELECT server,observation FROM autoscaling_members WHERE policy_id=?').all(policy.id) as { server: string; observation: string }[]; for (const member of members) { diff --git a/src/routes/autoscaling/+page.svelte b/src/routes/autoscaling/+page.svelte index 847f699..8d58c2c 100644 --- a/src/routes/autoscaling/+page.svelte +++ b/src/routes/autoscaling/+page.svelte @@ -73,8 +73,8 @@
{ event.preventDefault(); void submit('preview'); }} oninput={() => preview = null} onchange={() => preview = null}>
- - + + @@ -82,15 +82,16 @@
+ {#if settings.mapSlug === '*'}

Targets apply separately to every map, including all variants. Each map needs its own keys. Actions share one cooldown and run one at a time across the game.

{/if}
{#if preview}

Preview · {modeLabel(preview.policy.mode)}

-

Target: {preview.decision.desired} running lobbies. {preview.snapshot.keyedNames.length} saved lobby keys. {preview.snapshot.members.length} existing or pending lobbies.

+

Target: {preview.decision.desired} running lobbies{preview.policy.mapSlug === '*' ? ' across all maps' : ''}. {preview.snapshot.keyedNames.length} saved lobby keys. {preview.snapshot.members.length} existing or pending lobbies.

{preview.decision.action === 'wait' ? 'Wait' : `${preview.decision.action === 'create' ? 'Create' : 'Remove'} ${preview.decision.server}`}: {preview.decision.reason}

- {#if preview.policy.mode === 'create-only' || preview.policy.mode === 'create-destroy'}

Saving gives this policy exclusive control of the map's lobby count. Pause it before launching a competing collection or manual lifecycle action.

{/if} + {#if preview.policy.mode === 'create-only' || preview.policy.mode === 'create-destroy'}

Saving gives this policy exclusive control of the selected maps' lobby counts. Pause it before launching a competing collection or manual lifecycle action.

{/if} {#if preview.snapshot.members.length}
    {#each preview.snapshot.members as member}
  • {member.name} · {member.state}{member.blocker ? ` · ${member.blocker}` : ''}
    • {/each}
{/if} {#each [...preview.blockers, ...preview.snapshot.blockers] as blocker}

{blocker}

{/each} @@ -98,13 +99,13 @@ {/if} {/if} - {#if !report.policies.length}

No autoscaling policies. Add a policy to observe a map in dry run.

{/if} + {#if !report.policies.length}

No autoscaling policies. Add a policy to observe a map or a whole game in dry run.

{/if} {#each report.policies as view (view.policy.id)}
-

{zombieMapBySlug(view.policy.game, view.policy.mapSlug)?.friendly ?? view.policy.mapSlug} · {view.policy.game.toUpperCase()}

{modeLabel(view.policy.mode)}{#if canOperate}{/if}
+

{view.policy.mapSlug === '*' ? 'Every map' : zombieMapBySlug(view.policy.game, view.policy.mapSlug)?.friendly ?? view.policy.mapSlug} · {view.policy.game.toUpperCase()}

{modeLabel(view.policy.mode)}{#if canOperate}{/if}

{view.decision.occupied} occupied · {view.decision.empty} empty and ready · {view.decision.pending} pending · {view.decision.unknown} unknown or stopped

- {#if view.unsettled.length}{/if} -

Minimum {view.policy.minimum} · Maximum {view.policy.maximum} · Spare target {view.policy.spares} · {view.snapshot.keyedNames.length} saved keys

+ {#if view.unsettled.length}{/if} +

{view.policy.mapSlug === '*' ? 'Per map: ' : ''}Minimum {view.policy.minimum} · Maximum {view.policy.maximum} · Spare target {view.policy.spares} · {view.snapshot.keyedNames.length} saved keys

{view.policy.mode === 'dry-run' ? 'Hypothetical decision: ' : 'Current decision: '}{view.decision.action}{view.decision.server ? ` ${view.decision.server}` : ''}. {view.decision.reason}

{#if view.policy.mode === 'create-only' && view.decision.action === 'destroy'}

Removal is advisory in Create only mode.

{/if}

Last evaluated: {view.lastEvaluatedAt ? $timeFormat.dateTime(new Date(view.lastEvaluatedAt).toISOString()) : 'Not evaluated yet'}{view.policy.nextActionAt > Date.now() ? ` · Next action after ${$timeFormat.dateTime(new Date(view.policy.nextActionAt).toISOString())}` : ''}

diff --git a/tests/unit/autoscaling-game.test.ts b/tests/unit/autoscaling-game.test.ts new file mode 100644 index 0000000..85f599b --- /dev/null +++ b/tests/unit/autoscaling-game.test.ts @@ -0,0 +1,73 @@ +import { afterEach, beforeEach, expect, it, vi } from 'vitest'; +import { newFixture, type Fixture } from '../helpers/fixture.js'; +import { prepareNativeHost } from '../helpers/native-host.js'; +import { zombieMaps } from '../../src/lib/map-aliases.js'; +import { scalingSettings, type ScalingMember, type ScalingPolicy } from '../../src/lib/autoscaling.js'; +import { planScaling } from '../../src/lib/server/autoscaling/planner.js'; + +let fixture: Fixture; +beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); }); +afterEach(async () => { await fixture.cleanup(); }); +const settings = { game: 't6', mapSlug: '*', maximum: 2, mode: 'create-only' }; +const maps = zombieMaps('t6'); +async function setup() { + await prepareNativeHost(); + const { db } = await import('../../src/lib/server/db/index.js'); + const scaling = await import('../../src/lib/server/autoscaling/index.js'); + for (const map of maps) for (const n of [1, 2]) db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(`t6_zombie_${map.slug}_${n}`, `key-${map.slug}-${n}`); + return { db, scaling }; +} + +it('applies targets and spare deficits independently to every map', () => { + const now = 2_000_000; + const policy: ScalingPolicy = { ...scalingSettings(settings), id: 'policy', revision: 1, nextActionAt: 0, pausedReason: null, updatedAt: '' }; + const members: ScalingMember[] = maps.flatMap(map => [1, 2].map(n => ({ name: `t6_zombie_${map.slug}_${n}`, mapSlug: map.slug, + identity: `${map.slug}:${n}`, state: map.slug === 'town' ? 'occupied' : 'empty', players: map.slug === 'town' ? 1 : 0, + observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null }))); + const snapshot = { members, reservedNames: members.map(m => m.name), keyedNames: members.map(m => m.name), blockers: [] }; + // Other maps' empty lobbies cannot satisfy Town's spare target or maximum. + expect(planScaling(policy, snapshot, now)).toMatchObject({ action: 'destroy', server: 't6_zombie_transit_1', desired: maps.length + 1, deficit: 1 }); + snapshot.members = members.filter(m => m.name !== 't6_zombie_town_2'); + snapshot.reservedNames = snapshot.members.map(m => m.name); + expect(planScaling(policy, snapshot, now)).toMatchObject({ action: 'create', server: 't6_zombie_town_2' }); + expect(planScaling({ ...policy, nextActionAt: now + 1 }, snapshot, now).action).toBe('wait'); + snapshot.members[0] = { ...snapshot.members[0], state: 'creating' }; + expect(planScaling(policy, snapshot, now).action).toBe('wait'); +}); + +it('persists a game rule, queues the selected map recipe, and protects every map', async () => { + const { db, scaling } = await setup(); + db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run('t5_zombie_theater_1', 'other-game'); + const preview = scaling.previewScaling(settings); + expect(preview.snapshot.keyedNames).toHaveLength(maps.length * 2); + const saved = scaling.saveScaling(settings, undefined, undefined, preview.token); + expect(() => scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token)).toThrow(); + expect(() => scaling.previewScaling({ ...settings, mapSlug: 'town' }, saved.id, saved.revision)).toThrow('cannot change'); + expect(await scaling.runAutoscaling()).toBe(true); + const job = db().prepare('SELECT deployment_id,map_slug,server FROM server_creation_jobs').get() as { deployment_id: string; map_slug: string; server: string }; + expect(job).toMatchObject({ map_slug: 'transit', server: 't6_zombie_transit_1' }); + const { scalingAdmission } = await import('../../src/lib/server/autoscaling/admission.js'); + expect(scalingAdmission(job.deployment_id)).toBeNull(); + await scaling.runAutoscaling(); + expect(db().prepare('SELECT COUNT(*) AS n FROM server_creation_jobs').get()).toEqual({ n: 1 }); + const { scalingOwner, scalingOwnershipBlocker } = await import('../../src/lib/server/autoscaling/ownership.js'); + for (const map of maps) expect(scalingOwner('t6', map.slug)).toBe(saved.id); + expect(scalingOwner('t5', 'theater')).toBeNull(); + expect(scalingOwnershipBlocker('t6_zombie_farm_1')).toContain(saved.id); + expect(scaling.previewScaling({ ...settings, mapSlug: 'town' }).blockers.join(' ')).toContain('overlapping'); + const off = { ...settings, mode: 'off' }; + scaling.saveScaling(off, saved.id, saved.revision, scaling.previewScaling(off, saved.id, saved.revision).token); + expect(scalingOwner('t6', 'town')).toBeNull(); + const mapSettings = { ...settings, mapSlug: 'town' }; + const mapPolicy = scaling.saveScaling(mapSettings, undefined, undefined, scaling.previewScaling(mapSettings).token); + const current = scaling.getScalingPolicy(saved.id)!; + expect(scaling.previewScaling(settings, saved.id, current.revision).blockers.join(' ')).toContain(mapPolicy.id); +}); + +it('requires live key coverage on each map even when the game has enough keys in total', async () => { + const { db, scaling } = await setup(); + db().prepare("DELETE FROM server_keys WHERE server_id='t6_zombie_town_2'").run(); + const preview = scaling.previewScaling(settings); + expect(preview.blockers.join(' ')).toContain('Town Survival'); + expect(() => scaling.saveScaling(settings, undefined, undefined, preview.token)).toThrow('Town Survival'); +}); diff --git a/tests/unit/autoscaling-removal.test.ts b/tests/unit/autoscaling-removal.test.ts index 400abae..24566ad 100644 --- a/tests/unit/autoscaling-removal.test.ts +++ b/tests/unit/autoscaling-removal.test.ts @@ -1,5 +1,6 @@ import { afterEach, beforeEach, expect, it, vi } from 'vitest'; import { newFixture, type Fixture } from '../helpers/fixture.js'; +import { zombieMaps } from '../../src/lib/map-aliases.js'; import { prepareNativeHost } from '../helpers/native-host.js'; vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconGetPlayerCount: vi.fn(async () => 0) })); @@ -9,24 +10,30 @@ afterEach(async () => { vi.useRealTimers(); await fixture.cleanup(); vi.restoreA const names = ['t6_zombie_town_1', 't6_zombie_town_2']; const settings = { game: 't6', mapSlug: 'town', maximum: 2, mode: 'create-destroy', emptyMinutes: 1, cooldownMinutes: 1 }; -async function setup() { - const host = await prepareNativeHost(names); +async function setup(gameWide = false) { + const servers = gameWide ? [...names, ...zombieMaps('t6').filter(m => m.slug !== 'town').map(m => `t6_zombie_${m.slug}_1`)] : names; + const policySettings = { ...settings, mapSlug: gameWide ? '*' : settings.mapSlug }; + const host = await prepareNativeHost(servers); const { db } = await import('../../src/lib/server/db/index.js'); const scaling = await import('../../src/lib/server/autoscaling/index.js'); const authorization = await import('../../src/lib/server/autoscaling/authorization.js'); db().prepare("UPDATE ec_hosts SET inventory=json_set(inventory,'$.capabilities',json(?)) WHERE id=?").run(JSON.stringify([authorization.autoscalingRemovalCapability]), host.host.id); const base = Date.now(); vi.useFakeTimers({ toFake: ['Date'] }); vi.setSystemTime(base); - for (const name of names) { - db().prepare("INSERT INTO servers(name,game,map,map_slug,container_state,last_players) VALUES(?,'t6','zm_transit','town','running',0)").run(name); + for (const name of servers) { + const map = zombieMaps('t6').find(m => name === `t6_zombie_${m.slug}_1` || name === `t6_zombie_${m.slug}_2`)!; + db().prepare("INSERT INTO servers(name,game,map,map_slug,container_state,last_players) VALUES(?,'t6',?,?,'running',0)").run(name, map.canonical, map.slug); db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(name, `key-${name}`); } + if (gameWide) for (const map of zombieMaps('t6').filter(m => m.slug !== 'town')) { + db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(`t6_zombie_${map.slug}_2`, `key-${map.slug}-2`); + } const fresh = (offset: number) => { vi.setSystemTime(base + offset); const at = new Date().toISOString(); db().prepare('UPDATE servers SET container_checked_at=?,last_seen_at=?,last_checked_at=?').run(at, at, at); db().prepare('UPDATE ec_hosts SET last_heartbeat=?').run(at); }; fresh(0); - const saved = scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token); + const saved = scaling.saveScaling(policySettings, undefined, undefined, scaling.previewScaling(policySettings).token); for (const offset of [0, 30_000, 60_000]) { fresh(offset); await scaling.runAutoscaling(); } const intent = db().prepare('SELECT operation_id FROM autoscaling_decisions WHERE resolved=0').get() as { operation_id: string }; expect(intent?.operation_id).toBeTruthy(); @@ -103,3 +110,10 @@ it('rejects expired, changed revision, and newly protected removal authorization s.fresh(130_000); expect(s.authorization.claimRemovalAuthorization(s.hostId, command.id, command.payload).authorized).toBe(false); }); + +it('authorizes a game-wide removal without trimming any map below its targets', async () => { + const s = await setup(true); + expect(s.db().prepare("SELECT server FROM server_deployments WHERE id=?").get(s.operationId)).toEqual({ server: names[0] }); + const command = await s.dispatch(); + expect(s.authorization.claimRemovalAuthorization(s.hostId, command.id, command.payload).authorized).toBe(true); +}); diff --git a/tests/unit/autoscaling-snapshot.test.ts b/tests/unit/autoscaling-snapshot.test.ts index 4293c8e..0a2cb4e 100644 --- a/tests/unit/autoscaling-snapshot.test.ts +++ b/tests/unit/autoscaling-snapshot.test.ts @@ -20,6 +20,7 @@ it('separates variants, deduplicates pending identities, and initializes additiv expect(scalingSnapshot(policy).members.map(m => [m.name, m.state])).toEqual([[town, 'empty']]); db().prepare(`INSERT INTO server_deployments(id,server,action,state,stage,started_at,triggered_by) VALUES('create',?,'create','queued','Awaiting placement',?,'test')`).run(town, at); db().prepare(`INSERT INTO server_creation_jobs(deployment_id,server,game_mode,map_slug,expected_map) VALUES('create',?,'zombie','town','zm_transit')`).run(town); + expect(scalingSnapshot({ ...policy, mapSlug: '*' }).members.map(m => m.name).sort()).toEqual([farm, town]); const snapshot = scalingSnapshot(policy); expect(snapshot.members).toHaveLength(1); expect(snapshot.members[0].state).toBe('creating');