Compare commits
5
Commits
a61eccaed6
...
75b6a0a783
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
75b6a0a783 | ||
|
|
34648b34e0 | ||
|
|
285279ee6d | ||
|
|
4dbf704e6f | ||
|
|
bc744e2840 |
+13
-7
@@ -1,12 +1,12 @@
|
||||
# Lobby autoscaling
|
||||
|
||||
Open **Fleet → Autoscaling** to keep empty lobbies available for new groups. Policies apply to a specific game and map variant. A partly occupied Zombies lobby is not an empty spare, even when it still has player slots.
|
||||
Open **Fleet → Autoscaling** to keep empty lobbies available for new groups. Policies apply to one map variant or every map in a game. A game-wide policy applies the same minimum, maximum, and spare targets independently to each map, including all variants. A partly occupied Zombies lobby is not an empty spare, even when it still has player slots.
|
||||
|
||||
Autoscaling is opt-in. Installing the feature creates no policies and changes no lobby counts. Use a two-week dry run before enabling live actions.
|
||||
|
||||
## Configure a policy
|
||||
|
||||
Choose **Add policy**, select the game and map, and enter an explicit maximum. The initial editor selects Dry run, with a minimum of one running lobby, one empty spare, a 20-minute empty interval, and a five-minute action cooldown. Counts must be between 1 and 20; the minimum and spare target cannot exceed the maximum.
|
||||
Choose **Add policy**, select the game and scope, and enter an explicit maximum. The initial editor selects Dry run, with a minimum of one running lobby, one empty spare, a 20-minute empty interval, and a five-minute action cooldown. Counts must be between 1 and 20; the minimum and spare target cannot exceed the maximum.
|
||||
|
||||
Choose **Preview changes** to inspect the existing lobbies, saved key coverage, target, proposed action, and blockers. **Save reviewed policy** checks the policy revision and current fleet again. If the preview changed, review a new preview before saving.
|
||||
|
||||
@@ -17,13 +17,13 @@ Choose **Preview changes** to inspect the existing lobbies, saved key coverage,
|
||||
| Create only | Adds a lobby when eligible. Surplus removal remains advisory. |
|
||||
| Create and destroy | Adds lobbies and removes surplus lobbies after continuous empty evidence and all removal checks pass. |
|
||||
|
||||
Live activation requires saved Plutonium keys for enough exact lobby identities to support the configured maximum. Keys may be nonconsecutive: if only lobby 3 and lobby 8 are keyed, those identities can be used. Existing lobbies and unfinished operations must be healthy and settled before activation. Automatic removal also requires agent **1.7.0 or later** advertising `instance.autoscaling-remove` on each affected host.
|
||||
Creating a policy in any mode, including Dry run or Off, requires saved Plutonium keys for enough exact lobby identities to support the configured maximum on every selected map. Preview lists each shortage, and Save rechecks coverage transactionally. Non-Off edits also require coverage; an existing policy can always be turned Off if keys have been removed. Only supported lobby identities numbered 1–20 with nonblank saved keys count. Keys may be nonconsecutive: if only lobby 3 and lobby 8 are keyed, those identities can be used. Existing lobbies and unfinished operations must be healthy and settled before activation. Automatic removal also requires agent **1.7.0 or later** advertising `instance.autoscaling-remove` on each affected host.
|
||||
|
||||
The desired count is `min(maximum, max(minimum, occupied lobbies + empty spare target))`. The controller evaluates policies every 15 seconds and requests at most one unresolved operation per map. Create uses the existing startup concurrency, CPU, memory, library, port, and host admission checks. Lack of host capacity leaves one request waiting; it does not create additional requests every tick. Creates use current promoted launch files when provisioning begins.
|
||||
The desired count is `min(maximum, max(minimum, occupied lobbies + empty spare target))`. The controller evaluates policies every 15 seconds and requests at most one unresolved operation per policy. Game-wide policies share one cooldown and one outstanding operation across their maps, prioritizing additions before removals. Create uses the existing startup concurrency, CPU, memory, library, port, and host admission checks. Lack of host capacity leaves one request waiting; it does not create additional requests every tick. Creates use current promoted launch files when provisioning begins.
|
||||
|
||||
## Ownership and blockers
|
||||
|
||||
A live policy owns the map's lobby count. Competing collection launches and manual Create, Start, Stop, Rebuild, or Destroy requests are rejected with a policy link. Exact collection matching also checks managed maps absent from the collection. To hand control back, set the policy to Off and wait for unsettled operations to finish.
|
||||
A live policy owns the lobby counts for its selected maps. Overlapping live game-wide and map-specific policies are rejected; dry-run policies may overlap. Game-wide rules include every catalog map and require key coverage for the maximum on each map. Competing collection launches and manual Create, Start, Stop, Rebuild, or Destroy requests are rejected with a policy link. Exact collection matching also checks managed maps absent from the collection. To hand control back, set the policy to Off and wait for unsettled operations to finish.
|
||||
|
||||
Unknown or stale occupancy, stopped or unhealthy members, active betas, campaigns, maintenance, host drains, and unresolved recovery hold the policy's decisions. Autoscaling does not repair these conditions itself. The page and linked Provisioning operation explain the current blocker, including capacity and missing-key waits.
|
||||
|
||||
@@ -31,7 +31,7 @@ Current positive occupancy from either telemetry or RCON vetoes an empty claim.
|
||||
|
||||
## Pause and cancellation
|
||||
|
||||
Use **Pause all** to stop new decisions globally, or **Edit or pause → Off** for one map. A global pause retains each live policy's ownership; setting it Off releases ownership once its outstanding operation settles. Resuming the global switch preserves each policy's selected mode.
|
||||
Use **Pause all** to stop new decisions globally, or **Edit or pause → Off** for one policy. A global pause retains each live policy's ownership; setting it Off releases ownership once its outstanding operation settles. Resuming the global switch preserves each policy's selected mode.
|
||||
|
||||
Unsubmitted creates and queued removals are withdrawn where possible. Cancelling an autoscaling operation from Provisioning also turns its policy Off so it cannot immediately request a replacement.
|
||||
|
||||
@@ -41,6 +41,12 @@ The authorization claim is the cancellation boundary. A pause after that point c
|
||||
|
||||
Turning a policy Off does not recreate lobbies it removed or remove lobbies it created. Once operations settle, use Collections to restore a fixed lineup through the normal capacity and empty-server gates.
|
||||
|
||||
## Delete a policy
|
||||
|
||||
Choose **Delete policy** and confirm to remove its settings, decisions, and measurements. Existing lobbies, saved keys, and Provisioning history are kept. Deleting a live policy releases ownership without changing the lineup, and a new policy can reuse the same scope.
|
||||
|
||||
Deletion requires the current policy revision and no unsettled autoscaling operation. If work is outstanding, turn the policy Off and wait for its operation to settle before deleting. Delivered or uncertain host commands retain their policy and admission evidence until reconciliation finishes.
|
||||
|
||||
## Dry run and pilot
|
||||
|
||||
The production observation period and live pilot have not been run as part of implementation. Begin them on the operator-selected maps after deploying the controller and compatible agents.
|
||||
@@ -59,6 +65,6 @@ Measurement totals exclude collection gaps. Request-to-verified-readiness summar
|
||||
|
||||
`GET /api/autoscaling` returns policies, current evidence, recent decisions, measurements, and global pause state. Viewers may read it; mutations require an operator or administrator, using the existing authentication and origin checks.
|
||||
|
||||
`POST /api/autoscaling` accepts `preview`, `save`, or `global-pause`. Preview/save requests contain `settings`, plus `id` and `revision` when editing. Save also requires the preview `token`. Global pause uses a boolean `paused` field. Agent removal claims use a separate host-authenticated endpoint and are bound to the exact command, host, instance generation, policy revision, and expiry.
|
||||
`POST /api/autoscaling` accepts `preview`, `save`, `delete`, or `global-pause`. Delete requires `id` and the current `revision`. Preview/save requests contain `settings`, plus `id` and `revision` when editing. Use `mapSlug: '*'` for every map in the selected game; existing map-specific requests are unchanged. Scope cannot change when editing. Save also requires the preview `token`. Global pause uses a boolean `paused` field. Agent removal claims use a separate host-authenticated endpoint and are bound to the exact command, host, instance generation, policy revision, and expiry.
|
||||
|
||||
Shared contracts live in `src/lib/autoscaling.ts`; the evaluator, snapshots, persistence, ownership, and authorization live in `src/lib/server/autoscaling/`. Provisioning remains the only lifecycle executor. See the [approved implementation plan](lobby-autoscaling-plan.md) for the design and rollout criteria.
|
||||
|
||||
@@ -146,3 +146,9 @@ Validation includes planner and snapshot tests, duplicate-admission and stale-pr
|
||||
The full controller suite reported 1,934 passing tests and two existing failures in `server-detail-api.test.ts` (plugin classification) and `watchdog.test.ts` (an occupied-telemetry RCON expectation). Both failures reproduce independently at the pre-implementation commit `a22461a`. They are not autoscaling regressions. Targeted controller tests, all 228 Python agent tests, type checking, and the production build passed.
|
||||
|
||||
An isolated Firefox review exercised policy creation, preview, save, keyboard focus, and 1440, 390, and 320 pixel layouts. A narrow-layout heading issue found during review was corrected. The preview used a temporary database and no production host credentials. Production CPU savings, external joinability, spare targets, and acceptable deficit durations still require the two-week observation period and operator-run pilot.
|
||||
|
||||
Game-wide policy follow-up: selecting every map applies the targets independently to each catalog map and variant. Game-wide policies serialize actions across maps and cannot overlap live map-specific ownership. Existing map policies remain compatible.
|
||||
|
||||
Key coverage follow-up: creation in every mode requires sufficient saved keys for the maximum on each selected map. Coverage is rechecked on save, including edits to active or dry-run policies. Turning an existing policy Off remains available after key loss.
|
||||
|
||||
Follow-up validation: 40 autoscaling tests and 118 related provisioning, collection, and cancellation tests passed, along with type checking and the production build. An isolated Firefox review verified that missing per-map keys disable Save, complete coverage permits game-wide dry-run creation, and keyboard navigation and 1440, 390, and 320 pixel layouts work.
|
||||
|
||||
@@ -3,14 +3,17 @@ import { zombieMapBySlug, type ZombieGame } from './map-aliases.js';
|
||||
export const autoscalingModes = ['off', 'dry-run', 'create-only', 'create-destroy'] as const;
|
||||
export type AutoscalingMode = typeof autoscalingModes[number];
|
||||
export interface ScalingSettings {
|
||||
game: ZombieGame; mapSlug: string; mode: AutoscalingMode;
|
||||
game: ZombieGame;
|
||||
/** '*' applies the targets independently to every map. */
|
||||
mapSlug: string;
|
||||
mode: AutoscalingMode;
|
||||
minimum: number; maximum: number; spares: number; emptyMinutes: number; cooldownMinutes: number;
|
||||
}
|
||||
export interface ScalingPolicy extends ScalingSettings {
|
||||
id: string; revision: number; nextActionAt: number; pausedReason: string | null; updatedAt: string;
|
||||
}
|
||||
export interface ScalingMember {
|
||||
name: string; identity: string; state: 'occupied' | 'empty' | 'creating' | 'removing' | 'stopped' | 'unknown' | 'protected';
|
||||
name: string; mapSlug?: string; identity: string; state: 'occupied' | 'empty' | 'creating' | 'removing' | 'stopped' | 'unknown' | 'protected';
|
||||
players: number | null; observedAt: number | null; source: string | null;
|
||||
emptySince: number | null; blocker: string | null; operationId: string | null;
|
||||
emptyStartedAt?: number | null;
|
||||
@@ -20,7 +23,7 @@ export interface ScalingSnapshot {
|
||||
}
|
||||
export interface ScalingDecision {
|
||||
action: 'create' | 'destroy' | 'wait'; reason: string; server: string | null;
|
||||
desired: number; occupied: number; empty: number; pending: number; unknown: number;
|
||||
desired: number; deficit: number; occupied: number; empty: number; pending: number; unknown: number;
|
||||
}
|
||||
export interface ScalingPreview { policy: ScalingPolicy; snapshot: ScalingSnapshot; decision: ScalingDecision; token: string; blockers: string[] }
|
||||
export interface ScalingHistory { id: string; at: number; lastSeenAt: number; decision: ScalingDecision; operationId: string | null; operationState: string | null }
|
||||
@@ -38,7 +41,7 @@ export function scalingSettings(input: unknown): ScalingSettings {
|
||||
if (!input || typeof input !== 'object' || Array.isArray(input)) throw new Error('Provide an autoscaling policy.');
|
||||
const value = input as Record<string, unknown>;
|
||||
const { game, mapSlug } = value;
|
||||
if ((game !== 't5' && game !== 't6') || typeof mapSlug !== 'string' || zombieMapBySlug(game, mapSlug)?.slug !== mapSlug) throw new Error('Choose a supported game and map variant.');
|
||||
if ((game !== 't5' && game !== 't6') || typeof mapSlug !== 'string' || (mapSlug !== '*' && zombieMapBySlug(game, mapSlug)?.slug !== mapSlug)) throw new Error('Choose a supported game and map variant, or all maps.');
|
||||
const mode = value.mode ?? 'off';
|
||||
if (!autoscalingModes.includes(mode as AutoscalingMode)) throw new Error('Choose a valid autoscaling mode.');
|
||||
const integer = (key: string, fallback: number | undefined, max: number) => {
|
||||
|
||||
@@ -2,6 +2,7 @@ import { createHash, randomUUID } from 'node:crypto';
|
||||
import { db } from '../db/index.js';
|
||||
import { liveScaling, scalingSettings, type ScalingPolicy, type ScalingPreview, type ScalingReport, type ScalingView } from '../../autoscaling.js';
|
||||
import { emptyObservation, planScaling } from './planner.js';
|
||||
import { zombieMaps, parseServerName } from '../../map-aliases.js';
|
||||
import { scalingSnapshot } from './snapshot.js';
|
||||
import { nativeLaunchSelection, queueNativeCreationIntent } from '../provisioning/native.js';
|
||||
import { cancelCreation } from '../provisioning/cancel-creation.js';
|
||||
@@ -10,6 +11,7 @@ import { destroyServer } from '../provisioning/instances.js';
|
||||
import { cancelRemoval } from '../provisioning/history.js';
|
||||
import { autoscalingRemovalCapability, revokePolicyRemovals } from './authorization.js';
|
||||
import { getHost } from '../hosts/index.js';
|
||||
import { scalingOwner } from './ownership.js';
|
||||
import { getServer } from '../fleet/servers.js';
|
||||
|
||||
export class ScalingError extends Error { constructor(message: string, public status = 400) { super(message); } }
|
||||
@@ -28,8 +30,16 @@ export function previewScaling(input: unknown, id?: string, revision?: number):
|
||||
nextActionAt: existing?.nextActionAt ?? 0, pausedReason: null, updatedAt: existing?.updatedAt ?? '' };
|
||||
const snapshot = scalingSnapshot(policy);
|
||||
const blockers: string[] = [];
|
||||
// Creating any policy requires full coverage; existing policies must always be pausable.
|
||||
if (!existing || policy.mode !== 'off') {
|
||||
for (const map of zombieMaps(policy.game).filter(m => policy.mapSlug === '*' || m.slug === policy.mapSlug)) {
|
||||
const count = snapshot.keyedNames.filter(name => parseServerName(name)?.mapSlug === map.slug).length;
|
||||
if (count < policy.maximum) blockers.push(`Save keys for at least ${policy.maximum} lobby identities for ${map.friendly} before saving this policy (${count} saved; ${policy.maximum - count} more required).`);
|
||||
}
|
||||
}
|
||||
if (liveScaling(policy.mode)) {
|
||||
if (snapshot.keyedNames.length < policy.maximum) blockers.push(`Save keys for at least ${policy.maximum} lobby identities before activating this policy.`);
|
||||
const owner = scalingOwner(policy.game, policy.mapSlug);
|
||||
if (owner && owner !== policy.id) blockers.push(`An overlapping policy owns this lineup. Pause it and settle pending work first: /autoscaling#${owner}`);
|
||||
blockers.push(...snapshot.blockers);
|
||||
for (const member of snapshot.members) if (!['occupied', 'empty'].includes(member.state)) blockers.push(`${member.name}: ${member.blocker ?? member.state}. Settle this work before activation.`);
|
||||
if (policy.mode === 'create-destroy') for (const member of snapshot.members) {
|
||||
@@ -52,7 +62,7 @@ export function saveScaling(input: unknown, id?: string, revision?: number, toke
|
||||
if (preview.blockers.length) throw new ScalingError(preview.blockers[0]);
|
||||
const policy = preview.policy, settings = scalingSettings(policy), nextId = id ?? randomUUID(), at = new Date().toISOString();
|
||||
if (id) revokePolicyRemovals(id);
|
||||
if (!id && db().prepare('SELECT 1 FROM autoscaling_policies WHERE game=? AND map_slug=?').get(policy.game, policy.mapSlug)) throw new ScalingError('A policy already exists for this map.', 409);
|
||||
if (!id && db().prepare('SELECT 1 FROM autoscaling_policies WHERE game=? AND map_slug=?').get(policy.game, policy.mapSlug)) throw new ScalingError('A policy already exists for this scope.', 409);
|
||||
db().prepare(`INSERT INTO autoscaling_policies(id,game,map_slug,settings,mode,updated_at) VALUES(?,?,?,?,?,?)
|
||||
ON CONFLICT(id) DO UPDATE SET settings=excluded.settings,mode=excluded.mode,revision=autoscaling_policies.revision+1,paused_reason=NULL,updated_at=excluded.updated_at`).run(nextId, policy.game, policy.mapSlug, JSON.stringify(settings), policy.mode, at);
|
||||
// An edit starts a new observation interval; earlier inactivity is not authorization.
|
||||
@@ -65,6 +75,24 @@ export function saveScaling(input: unknown, id?: string, revision?: number, toke
|
||||
})();
|
||||
}
|
||||
|
||||
export function deleteScaling(id: string, revision: number): void {
|
||||
db().transaction(() => {
|
||||
const policy = getScalingPolicy(id);
|
||||
if (!policy) throw new ScalingError('Policy not found.', 404);
|
||||
if (policy.revision !== revision) throw new ScalingError('Policy changed. Refresh before deleting.', 409);
|
||||
// Keep admission evidence and ownership while a host command may still act.
|
||||
if (db().prepare('SELECT 1 FROM autoscaling_decisions WHERE policy_id=? AND resolved=0').get(id)) {
|
||||
throw new ScalingError('Pause this policy and settle its outstanding Provisioning work before deleting it.', 409);
|
||||
}
|
||||
revokePolicyRemovals(id);
|
||||
db().prepare('DELETE FROM autoscaling_authorizations WHERE policy_id=?').run(id);
|
||||
db().prepare('DELETE FROM autoscaling_members WHERE policy_id=?').run(id);
|
||||
db().prepare('DELETE FROM autoscaling_samples WHERE policy_id=?').run(id);
|
||||
db().prepare('DELETE FROM autoscaling_decisions WHERE policy_id=?').run(id);
|
||||
db().prepare('DELETE FROM autoscaling_policies WHERE id=?').run(id);
|
||||
})();
|
||||
}
|
||||
|
||||
export function setScalingGlobalPause(paused: boolean): void {
|
||||
db().transaction(() => {
|
||||
db().prepare("INSERT INTO settings(key,value) VALUES('autoscaling_paused',?) ON CONFLICT(key) DO UPDATE SET value=excluded.value").run(paused ? '1' : '0');
|
||||
@@ -139,7 +167,7 @@ export async function runAutoscaling(): Promise<boolean> {
|
||||
try {
|
||||
db().transaction(() => {
|
||||
const result = decision.action === 'create'
|
||||
? queueNativeCreationIntent({ server: decision.server!, game: policy.game, mapSlug: policy.mapSlug,
|
||||
? queueNativeCreationIntent({ server: decision.server!, game: policy.game, mapSlug: parseServerName(decision.server!)!.mapSlug,
|
||||
selection: nativeLaunchSelection({ automatic: true }, policy.game), createdBy: `autoscaling:${policy.id}`, autoscalingDecisionId: decisionId })
|
||||
: destroyServer(decision.server!, `autoscaling:${policy.id}`, decisionId);
|
||||
if (!result.ok) throw new Error(result.message);
|
||||
@@ -155,7 +183,7 @@ export async function runAutoscaling(): Promise<boolean> {
|
||||
}
|
||||
}
|
||||
db().prepare(`INSERT OR IGNORE INTO autoscaling_samples(policy_id,sampled_at,mode,occupied,empty,unknown,deficit) VALUES(?,?,?,?,?,?,?)`)
|
||||
.run(policy.id, Math.floor(now / 15_000) * 15_000, policy.mode, decision.occupied, decision.empty, decision.unknown, Math.max(0, policy.spares - decision.empty));
|
||||
.run(policy.id, Math.floor(now / 15_000) * 15_000, policy.mode, decision.occupied, decision.empty, decision.unknown, decision.deficit);
|
||||
})();
|
||||
}
|
||||
return submitted;
|
||||
|
||||
@@ -3,7 +3,7 @@ import { getServer } from '../fleet/servers.js';
|
||||
import { mapIdentity } from './snapshot.js';
|
||||
|
||||
export function scalingOwner(game: string, mapSlug: string): string | null {
|
||||
return (db().prepare('SELECT id FROM autoscaling_policies WHERE game=? AND map_slug=? AND owns_map=1').get(game, mapSlug) as { id: string } | undefined)?.id ?? null;
|
||||
return (db().prepare("SELECT id FROM autoscaling_policies WHERE game=? AND (map_slug=? OR map_slug='*' OR ?='*') AND owns_map=1").get(game, mapSlug, mapSlug) as { id: string } | undefined)?.id ?? null;
|
||||
}
|
||||
export function scalingOwnershipBlocker(server: string): string | null {
|
||||
const identity = mapIdentity(getServer(server) ?? { name: server });
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import type { ScalingDecision, ScalingMember, ScalingPolicy, ScalingSnapshot } from '../../autoscaling.js';
|
||||
import { parseServerName, zombieMaps } from '../../map-aliases.js';
|
||||
import { ONLINE_MAX_MS } from '../../server-freshness.js';
|
||||
|
||||
export interface EmptyObservation { identity: string; observedAt: number | null; emptySince: number | null; evaluatedAt: number }
|
||||
@@ -14,13 +15,32 @@ export function emptyObservation(member: ScalingMember, previous: EmptyObservati
|
||||
}
|
||||
|
||||
export function planScaling(policy: ScalingPolicy, snapshot: ScalingSnapshot, now: number): ScalingDecision {
|
||||
if (policy.mapSlug === '*') {
|
||||
const maps = zombieMaps(policy.game);
|
||||
const decisions = maps.map(map => {
|
||||
const matches = (name: string) => (snapshot.members.find(m => m.name === name)?.mapSlug ?? parseServerName(name)?.mapSlug) === map.slug;
|
||||
return planScaling({ ...policy, mapSlug: map.slug }, {
|
||||
...snapshot, members: snapshot.members.filter(m => matches(m.name)),
|
||||
reservedNames: snapshot.reservedNames.filter(matches), keyedNames: snapshot.keyedNames.filter(matches)
|
||||
}, now);
|
||||
});
|
||||
// Serialize game-wide work and prefer replenishing any map before trimming another.
|
||||
const pending = snapshot.members.find(m => m.state === 'creating' || m.state === 'removing');
|
||||
const selected = decisions.find(d => d.action === 'create') ?? decisions.find(d => d.action === 'destroy')
|
||||
?? decisions.find(d => d.deficit > 0 || d.unknown > 0) ?? decisions[0];
|
||||
const map = maps[decisions.indexOf(selected)];
|
||||
return { ...selected, action: pending ? 'wait' : selected.action, server: pending ? null : selected.server,
|
||||
reason: pending ? pending.blocker ?? `Waiting for ${pending.name}: ${pending.state}.` : `${map.friendly}: ${selected.reason}`,
|
||||
desired: decisions.reduce((n, d) => n + d.desired, 0), deficit: decisions.reduce((n, d) => n + d.deficit, 0), occupied: decisions.reduce((n, d) => n + d.occupied, 0),
|
||||
empty: decisions.reduce((n, d) => n + d.empty, 0), pending: decisions.reduce((n, d) => n + d.pending, 0), unknown: decisions.reduce((n, d) => n + d.unknown, 0) };
|
||||
}
|
||||
const { members } = snapshot;
|
||||
const occupied = members.filter(m => m.state === 'occupied').length, empty = members.filter(m => m.state === 'empty').length;
|
||||
const pending = members.filter(m => m.state === 'creating' || m.state === 'removing').length;
|
||||
const unknown = members.filter(m => m.state === 'unknown' || m.state === 'stopped').length;
|
||||
const desired = Math.min(policy.maximum, Math.max(policy.minimum, occupied + policy.spares));
|
||||
const result = (action: ScalingDecision['action'], reason: string, server: string | null = null): ScalingDecision =>
|
||||
({ action, reason, server, desired, occupied, empty, pending, unknown });
|
||||
({ action, reason, server, desired, deficit: Math.max(0, policy.spares - empty), occupied, empty, pending, unknown });
|
||||
if (policy.mode === 'off') return result('wait', policy.pausedReason ?? 'Policy is off.');
|
||||
if (policy.pausedReason) return result('wait', policy.pausedReason);
|
||||
if (snapshot.blockers.length) return result('wait', snapshot.blockers[0]);
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { db } from '../db/index.js';
|
||||
import { validServerId } from '../fleet/server-keys.js';
|
||||
import { listServers, type ServerRow } from '../fleet/servers.js';
|
||||
import { observationReader } from '../fleet/status.js';
|
||||
import { parseServerName, zombieMaps } from '../../map-aliases.js';
|
||||
@@ -28,7 +29,7 @@ export function protectedWork(server: string, exceptOperation = ''): string | nu
|
||||
export function scalingSnapshot(policy: ScalingPolicy, now = Date.now(), exceptOperation = ''): ScalingSnapshot {
|
||||
const handle = db(), observe = observationReader();
|
||||
const rows = listServers();
|
||||
const matches = (value: Parameters<typeof mapIdentity>[0]) => { const identity = mapIdentity(value); return identity?.game === policy.game && identity.mapSlug === policy.mapSlug; };
|
||||
const matches = (value: Parameters<typeof mapIdentity>[0]) => { const identity = mapIdentity(value); return identity?.game === policy.game && (policy.mapSlug === '*' || identity.mapSlug === policy.mapSlug); };
|
||||
const blockers: string[] = [];
|
||||
for (const row of rows) if (!mapIdentity(row) && (row.game ?? parseServerName(row.name)?.game) === policy.game) blockers.push(`Resolve ambiguous map identity for ${row.name}.`);
|
||||
const operations = handle.prepare(`SELECT id,server,action,state,stage FROM server_deployments WHERE server_deleted_at IS NULL
|
||||
@@ -54,12 +55,12 @@ export function scalingSnapshot(policy: ScalingPolicy, now = Date.now(), exceptO
|
||||
else if (row.desired_state === 'stopped' || ['stopped','exited','created','dead'].includes(row.container_state ?? '')) state = 'stopped';
|
||||
else if (observation.health === 'online' && players !== null && observedAt !== null && observedAt <= now && freshStatusTimestamp(new Date(observedAt).toISOString(), now)) state = players > 0 ? 'occupied' : 'empty';
|
||||
const transitions = [row.inactive_since, observation.inactive_since].map(at => Date.parse(at ?? '')).filter(at => Number.isFinite(at) && at <= now);
|
||||
return { name: row.name, identity, state, players, observedAt, source: source ?? null, emptySince: null,
|
||||
return { name: row.name, mapSlug: mapIdentity(row)!.mapSlug, identity, state, players, observedAt, source: source ?? null, emptySince: null,
|
||||
emptyStartedAt: transitions.length ? Math.max(...transitions) : null, blocker, operationId: operation?.id ?? null };
|
||||
});
|
||||
for (const row of reservations.filter(matches)) if (!members.some(m => m.name === row.name)) {
|
||||
const operation = operations.find(o => o.server === row.name);
|
||||
members.push({ name: row.name, identity: 'reserved', state: 'creating', players: null, observedAt: null, source: null, emptySince: null, blocker: operation?.stage ?? 'Creation is pending.', operationId: operation?.id ?? null });
|
||||
members.push({ name: row.name, mapSlug: row.map_slug, identity: 'reserved', state: 'creating', players: null, observedAt: null, source: null, emptySince: null, blocker: operation?.stage ?? 'Creation is pending.', operationId: operation?.id ?? null });
|
||||
}
|
||||
const retained = handle.prepare('SELECT server,observation FROM autoscaling_members WHERE policy_id=?').all(policy.id) as { server: string; observation: string }[];
|
||||
for (const member of members) {
|
||||
@@ -72,6 +73,6 @@ export function scalingSnapshot(policy: ScalingPolicy, now = Date.now(), exceptO
|
||||
reservedNames.add(row.name);
|
||||
if (!members.some(m => m.name === row.name)) blockers.push(`Unresolved native instance ${row.name}.`);
|
||||
}
|
||||
const keyedNames = (handle.prepare("SELECT server_id AS name FROM server_keys WHERE length(trim(pluto_key))>0").all() as { name: string }[]).filter(matches).map(r => r.name);
|
||||
const keyedNames = (handle.prepare("SELECT server_id AS name FROM server_keys WHERE length(trim(pluto_key))>0").all() as { name: string }[]).filter(row => validServerId(row.name) && matches(row)).map(r => r.name);
|
||||
return { members, reservedNames: [...reservedNames], keyedNames, blockers };
|
||||
}
|
||||
|
||||
@@ -60,7 +60,10 @@ function warningMessages(raw: string): string[] {
|
||||
|
||||
function sourceDetails(source: Omit<ProvisioningSource, 'label' | 'href'>): ProvisioningSource {
|
||||
const id = source.source_id;
|
||||
if (source.kind === 'autoscaling') return { ...source, label: 'Autoscaling policy', href: `/autoscaling#${encodeURIComponent(id)}` };
|
||||
if (source.kind === 'autoscaling') {
|
||||
const exists = !!db().prepare('SELECT 1 FROM autoscaling_policies WHERE id=?').get(id);
|
||||
return { ...source, label: exists ? 'Autoscaling policy' : 'Deleted autoscaling policy', href: exists ? `/autoscaling#${encodeURIComponent(id)}` : null };
|
||||
}
|
||||
if (source.kind === 'campaign') {
|
||||
const row = db().prepare(`SELECT d.action, d.beta_id, s.name FROM deployments d
|
||||
LEFT JOIN scripts s ON s.id=COALESCE(d.script_id, (SELECT script_id FROM deployment_scripts WHERE deployment_id=d.id ORDER BY script_id LIMIT 1))
|
||||
|
||||
@@ -143,7 +143,6 @@
|
||||
<section class="page">
|
||||
<SectionHeader title="Fleet" subtitle="Live health, players, and server operations.">
|
||||
{#snippet actions()}
|
||||
<Button href="/autoscaling" variant="ghost" size="sm">Autoscaling</Button>
|
||||
<Button variant="ghost" size="sm" loading={$fleetRefreshing} onclick={() => void refreshFleet()}>Refresh</Button>
|
||||
{#if canOperate}<Button variant="primary" onclick={() => launch()}>Create server</Button>{/if}
|
||||
{/snippet}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { json } from '@sveltejs/kit';
|
||||
import { previewScaling, saveScaling, scalingReport, setScalingGlobalPause, ScalingError } from '$lib/server/autoscaling/index.js';
|
||||
import { previewScaling, saveScaling, deleteScaling, scalingReport, setScalingGlobalPause, ScalingError } from '$lib/server/autoscaling/index.js';
|
||||
|
||||
export function GET() { return json(scalingReport()); }
|
||||
export async function POST({ request }: { request: Request }) {
|
||||
@@ -9,6 +9,11 @@ export async function POST({ request }: { request: Request }) {
|
||||
if (body.id !== undefined && typeof body.id !== 'string') throw new ScalingError('Invalid policy identity.');
|
||||
if (body.action === 'preview') return json(previewScaling(body.settings, body.id, body.revision));
|
||||
if (body.action === 'save') return json({ policy: saveScaling(body.settings, body.id, body.revision, body.token) });
|
||||
if (body.action === 'delete') {
|
||||
if (typeof body.id !== 'string' || !body.id || !Number.isInteger(body.revision) || body.revision < 1) throw new ScalingError('Provide the policy identity and current revision.');
|
||||
deleteScaling(body.id, body.revision);
|
||||
return json({ ok: true });
|
||||
}
|
||||
if (body.action === 'global-pause' && typeof body.paused === 'boolean') { setScalingGlobalPause(body.paused); return json({ ok: true }); }
|
||||
throw new ScalingError('Choose a supported policy action.');
|
||||
} catch (error) { return json({ error: (error as Error).message }, { status: error instanceof ScalingError ? error.status : 400 }); }
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
<script lang="ts">
|
||||
import { onMount } from 'svelte';
|
||||
import { page } from '$app/state';
|
||||
import { confirmAction } from '$ui/confirm.js';
|
||||
import { pollVisible } from '$ui/poll.js';
|
||||
import { timeFormat } from '$ui/stores/time-zone.js';
|
||||
import SectionHeader from '$ui/components/primitives/SectionHeader.svelte';
|
||||
@@ -48,6 +49,19 @@
|
||||
} catch (error) { actionError = (error as Error).message; preview = null; }
|
||||
finally { busy = false; }
|
||||
}
|
||||
async function remove(policy: ScalingPolicy) {
|
||||
const scope = policy.mapSlug === '*' ? `every map in ${policy.game.toUpperCase()}` : `${zombieMapBySlug(policy.game, policy.mapSlug)?.friendly ?? policy.mapSlug} · ${policy.game.toUpperCase()}`;
|
||||
if (!await confirmAction({ title: 'Delete autoscaling policy?', message: `Delete the policy for ${scope} and its scaling history? Existing lobbies, saved keys, and Provisioning history will be kept.`, confirmLabel: 'Delete policy', danger: true })) return;
|
||||
busy = true; actionError = ''; notice = ''; ++generation;
|
||||
try {
|
||||
await post({ action: 'delete', id: policy.id, revision: policy.revision });
|
||||
if (editing?.id === policy.id) { editing = null; editorOpen = false; preview = null; }
|
||||
snapshot = { ...report, policies: report.policies.filter(view => view.policy.id !== policy.id) };
|
||||
notice = 'Policy deleted.';
|
||||
await refresh();
|
||||
} catch (error) { actionError = (error as Error).message; }
|
||||
finally { busy = false; }
|
||||
}
|
||||
async function pauseAll() {
|
||||
busy = true; actionError = ''; ++generation;
|
||||
try { await post({ action: 'global-pause', paused: !report.globalPaused }); notice = report.globalPaused ? 'Autoscaling resumed.' : 'Autoscaling paused. Review any unsettled operations below.'; await refresh(); }
|
||||
@@ -61,9 +75,7 @@
|
||||
<SectionHeader title="Autoscaling" subtitle="Keep empty lobbies available for new groups, within the limits you choose.">
|
||||
{#snippet actions()}<Button onclick={() => refresh()}>Refresh</Button>{#if canOperate}<Button disabled={busy} onclick={pauseAll}>{report.globalPaused ? 'Resume autoscaling' : 'Pause all'}</Button><Button variant="primary" disabled={busy} onclick={() => edit(null)}>Add policy</Button>{/if}{/snippet}
|
||||
</SectionHeader>
|
||||
<p class="context"><a href="/">Fleet</a> · <a href="/collections">Collections</a> · <a href="/metrics">Metrics</a></p>
|
||||
{#if report.globalPaused}<p class="banner">All autoscaling decisions are paused. Delivered operations may still need to settle.</p>{/if}
|
||||
<p>Start with a two-week dry run. Dry run records hypothetical actions and reserves no lobby or host capacity.</p>
|
||||
{#if refreshError}<p role="alert">{refreshError}</p>{/if}
|
||||
{#if actionError}<p role="alert">{actionError}</p>{/if}
|
||||
{#if notice}<p role="status">{notice}</p>{/if}
|
||||
@@ -73,8 +85,8 @@
|
||||
<form onsubmit={event => { event.preventDefault(); void submit('preview'); }} oninput={() => preview = null} onchange={() => preview = null}>
|
||||
<fieldset disabled={busy}>
|
||||
<div class="fields">
|
||||
<label>Game<select bind:value={settings.game} disabled={!!editing} onchange={() => settings.mapSlug = zombieMaps(settings.game)[0].slug}><option value="t5">Black Ops I</option><option value="t6">Black Ops II</option></select></label>
|
||||
<label>Map variant<select bind:value={settings.mapSlug} disabled={!!editing}>{#each zombieMaps(settings.game) as map}<option value={map.slug}>{map.friendly}</option>{/each}</select></label>
|
||||
<label>Game<select bind:value={settings.game} disabled={!!editing} onchange={() => { if (settings.mapSlug !== '*') settings.mapSlug = zombieMaps(settings.game)[0].slug; }}><option value="t5">Black Ops I</option><option value="t6">Black Ops II</option></select></label>
|
||||
<label>Scope<select bind:value={settings.mapSlug} disabled={!!editing}><option value="*">Every map in this game</option>{#each zombieMaps(settings.game) as map}<option value={map.slug}>{map.friendly}</option>{/each}</select></label>
|
||||
<label>Mode<select bind:value={settings.mode}><option value="off">Off</option><option value="dry-run">Dry run</option>{#if report.liveAvailable}<option value="create-only">Create only</option>{/if}{#if report.removalAvailable}<option value="create-destroy">Create and destroy</option>{/if}</select></label>
|
||||
<label>Minimum running lobbies<input type="number" min="1" max="20" required bind:value={settings.minimum} /></label>
|
||||
<label>Maximum lobbies<input type="number" min="1" max="20" required bind:value={settings.maximum} /></label>
|
||||
@@ -82,15 +94,17 @@
|
||||
<label>Empty duration in minutes<input type="number" min="1" max="1440" required bind:value={settings.emptyMinutes} /></label>
|
||||
<label>Action cooldown in minutes<input type="number" min="1" max="1440" required bind:value={settings.cooldownMinutes} /></label>
|
||||
</div>
|
||||
{#if settings.mapSlug === '*'}<p>Targets apply separately to every map, including all variants. Each map needs its own keys. Actions share one cooldown and run one at a time across the game.</p>{/if}
|
||||
<p>Saved lobby keys must cover the maximum for each selected map before you can save a new policy, including Dry run.</p>
|
||||
<div class="actions"><Button type="submit" disabled={busy}>Preview changes</Button><Button onclick={() => editorOpen = false}>Close editor</Button></div>
|
||||
</fieldset>
|
||||
</form>
|
||||
{#if preview}
|
||||
<div class="preview" aria-live="polite">
|
||||
<h3>Preview · {modeLabel(preview.policy.mode)}</h3>
|
||||
<p>Target: {preview.decision.desired} running lobbies. {preview.snapshot.keyedNames.length} saved lobby keys. {preview.snapshot.members.length} existing or pending lobbies.</p>
|
||||
<p>Target: {preview.decision.desired} running lobbies{preview.policy.mapSlug === '*' ? ' across all maps' : ''}. {preview.snapshot.keyedNames.length} saved lobby keys. {preview.snapshot.members.length} existing or pending lobbies.</p>
|
||||
<p>{preview.decision.action === 'wait' ? 'Wait' : `${preview.decision.action === 'create' ? 'Create' : 'Remove'} ${preview.decision.server}`}: {preview.decision.reason}</p>
|
||||
{#if preview.policy.mode === 'create-only' || preview.policy.mode === 'create-destroy'}<p>Saving gives this policy exclusive control of the map's lobby count. Pause it before launching a competing collection or manual lifecycle action.</p>{/if}
|
||||
{#if preview.policy.mode === 'create-only' || preview.policy.mode === 'create-destroy'}<p>Saving gives this policy exclusive control of the selected maps' lobby counts. Pause it before launching a competing collection or manual lifecycle action.</p>{/if}
|
||||
{#if preview.snapshot.members.length}<ul>{#each preview.snapshot.members as member}<li>{member.name} · {member.state}{member.blocker ? ` · ${member.blocker}` : ''}</li>{/each}</ul>{/if}
|
||||
{#each [...preview.blockers, ...preview.snapshot.blockers] as blocker}<p>{blocker}</p>{/each}
|
||||
<Button variant="primary" disabled={busy || preview.blockers.length > 0} onclick={() => submit('save')}>Save reviewed policy</Button>
|
||||
@@ -98,13 +112,13 @@
|
||||
{/if}
|
||||
</section>
|
||||
{/if}
|
||||
{#if !report.policies.length}<p class="panel">No autoscaling policies. Add a policy to observe a map in dry run.</p>{/if}
|
||||
{#if !report.policies.length}<p class="panel">No autoscaling policies. Add a policy to observe a map or a whole game in dry run.</p>{/if}
|
||||
{#each report.policies as view (view.policy.id)}
|
||||
<section class="panel" id={view.policy.id} aria-labelledby={`title-${view.policy.id}`}>
|
||||
<div class="heading"><h2 id={`title-${view.policy.id}`}>{zombieMapBySlug(view.policy.game, view.policy.mapSlug)?.friendly ?? view.policy.mapSlug} · {view.policy.game.toUpperCase()}</h2><span>{modeLabel(view.policy.mode)}</span>{#if canOperate}<Button disabled={busy} onclick={() => edit(view.policy)}>Edit or pause</Button>{/if}</div>
|
||||
<div class="heading"><h2 id={`title-${view.policy.id}`}>{view.policy.mapSlug === '*' ? 'Every map' : zombieMapBySlug(view.policy.game, view.policy.mapSlug)?.friendly ?? view.policy.mapSlug} · {view.policy.game.toUpperCase()}</h2><span>{modeLabel(view.policy.mode)}</span>{#if canOperate}<Button disabled={busy} onclick={() => edit(view.policy)}>Edit or pause</Button><Button variant="danger" disabled={busy || view.unsettled.length > 0} title={view.unsettled.length ? 'Pause the policy and settle outstanding work before deleting it.' : undefined} onclick={() => remove(view.policy)}>Delete policy</Button>{/if}</div>
|
||||
<p class="counts">{view.decision.occupied} occupied · {view.decision.empty} empty and ready · {view.decision.pending} pending · {view.decision.unknown} unknown or stopped</p>
|
||||
{#if view.unsettled.length}<div class="banner"><strong>Work is still settling.</strong>{#each view.unsettled as operation}<p><a href={`/provisioning/${operation.id}`}>{operation.action} · {operation.state}</a> — {operation.stage}</p>{/each}{#if view.ownsMap}<p>This policy retains map ownership until its outstanding work settles.</p>{/if}</div>{/if}
|
||||
<p>Minimum {view.policy.minimum} · Maximum {view.policy.maximum} · Spare target {view.policy.spares} · {view.snapshot.keyedNames.length} saved keys</p>
|
||||
{#if view.unsettled.length}<div class="banner"><strong>Work is still settling.</strong><p>Pause this policy and settle its outstanding work before deleting it.</p>{#each view.unsettled as operation}<p><a href={`/provisioning/${operation.id}`}>{operation.action} · {operation.state}</a> — {operation.stage}</p>{/each}{#if view.ownsMap}<p>This policy retains ownership of its scope until its outstanding work settles.</p>{/if}</div>{/if}
|
||||
<p>{view.policy.mapSlug === '*' ? 'Per map: ' : ''}Minimum {view.policy.minimum} · Maximum {view.policy.maximum} · Spare target {view.policy.spares} · {view.snapshot.keyedNames.length} saved keys</p>
|
||||
<p><strong>{view.policy.mode === 'dry-run' ? 'Hypothetical decision: ' : 'Current decision: '}</strong>{view.decision.action}{view.decision.server ? ` ${view.decision.server}` : ''}. {view.decision.reason}</p>
|
||||
{#if view.policy.mode === 'create-only' && view.decision.action === 'destroy'}<p>Removal is advisory in Create only mode.</p>{/if}
|
||||
<p class="muted">Last evaluated: {view.lastEvaluatedAt ? $timeFormat.dateTime(new Date(view.lastEvaluatedAt).toISOString()) : 'Not evaluated yet'}{view.policy.nextActionAt > Date.now() ? ` · Next action after ${$timeFormat.dateTime(new Date(view.policy.nextActionAt).toISOString())}` : ''}</p>
|
||||
@@ -125,7 +139,7 @@
|
||||
.panel { margin: 20px 0; padding: 20px; border: 1px solid var(--border-subtle); border-radius: var(--radius-lg); background: var(--surface); overflow-wrap: anywhere; }
|
||||
.heading, .actions { display: flex; gap: 12px; align-items: center; flex-wrap: wrap; }
|
||||
h2 { font-size: var(--text-lg); margin: 0; } h3 { font-size: var(--text-md); }
|
||||
.heading h2 { flex: 1 1 200px; } p, li { line-height: 1.6; } .muted, .context { color: var(--text-secondary); font-size: var(--text-sm); }
|
||||
.heading h2 { flex: 1 1 200px; } p, li { line-height: 1.6; } .muted { color: var(--text-secondary); font-size: var(--text-sm); }
|
||||
.fields { display: grid; grid-template-columns: repeat(auto-fit, minmax(min(100%, 220px), 1fr)); gap: 16px; margin-bottom: 20px; }
|
||||
label { display: flex; flex-direction: column; gap: 8px; font-size: var(--text-sm); }
|
||||
input, select { padding: 10px; min-width: 0; color: var(--text-primary); background: var(--surface); border: 1px solid var(--border-subtle); border-radius: var(--radius-md); font: inherit; }
|
||||
|
||||
@@ -94,7 +94,7 @@
|
||||
<svelte:head><title>Collections | Electric Cherry</title></svelte:head>
|
||||
<div class="page">
|
||||
<SectionHeader title="Collections" subtitle="Save your lineup, preview fleet changes, and follow launch progress and history.">
|
||||
{#snippet actions()}<Button href="/autoscaling">Autoscaling</Button>{#if canOperate}<Button variant="primary" onclick={() => { editingBlueprint = null; builderOpen = true; }}>Create collection</Button>{/if}{/snippet}
|
||||
{#snippet actions()}{#if canOperate}<Button variant="primary" onclick={() => { editingBlueprint = null; builderOpen = true; }}>Create collection</Button>{/if}{/snippet}
|
||||
</SectionHeader>
|
||||
{#if canOperate}<BlueprintBuilder bind:open={builderOpen} blueprint={editingBlueprint} onsaved={blueprintSaved} />
|
||||
<DeployCollectionDialog bind:open={collectionOpen} {blueprintId} />{/if}
|
||||
|
||||
@@ -108,7 +108,7 @@
|
||||
<svelte:head><title>Metrics · Electric Cherry</title></svelte:head>
|
||||
|
||||
<div class="page">
|
||||
<SectionHeader title="Metrics">{#snippet actions()}<Button href="/autoscaling">Autoscaling</Button>{/snippet}</SectionHeader>
|
||||
<SectionHeader title="Metrics" />
|
||||
<div class="toolbar">
|
||||
<nav aria-label="Metrics time range">
|
||||
{#each [1, 7, 30] as days}
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
import { afterEach, beforeEach, expect, it, vi } from 'vitest';
|
||||
import { newFixture, type Fixture } from '../helpers/fixture.js';
|
||||
import { prepareNativeHost } from '../helpers/native-host.js';
|
||||
|
||||
let fixture: Fixture;
|
||||
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
||||
afterEach(async () => { await fixture.cleanup(); });
|
||||
const settings = { game: 't6', mapSlug: 'town', maximum: 1, mode: 'dry-run' };
|
||||
async function setup(mode = 'dry-run') {
|
||||
if (mode.startsWith('create')) await prepareNativeHost();
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||
db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run('t6_zombie_town_1', 'test-key');
|
||||
const input = { ...settings, mode };
|
||||
const saved = scaling.saveScaling(input, undefined, undefined, scaling.previewScaling(input).token);
|
||||
return { db, scaling, saved, input };
|
||||
}
|
||||
|
||||
it('deletes policy evidence atomically while preserving lobbies, keys, and unrelated policies', async () => {
|
||||
const { db, scaling, saved } = await setup();
|
||||
db().prepare("INSERT INTO servers(name,game,map_slug,map) VALUES('t6_zombie_town_1','t6','town','zm_transit')").run();
|
||||
await scaling.runAutoscaling();
|
||||
db().prepare("INSERT INTO server_keys(server_id,pluto_key) VALUES('t5_zombie_theater_1','other-key')").run();
|
||||
const otherSettings = { ...settings, game: 't5', mapSlug: 'theater', mode: 'off' };
|
||||
const other = scaling.saveScaling(otherSettings, undefined, undefined, scaling.previewScaling(otherSettings).token);
|
||||
for (const table of ['autoscaling_decisions', 'autoscaling_members', 'autoscaling_samples']) expect(db().prepare(`SELECT COUNT(*) AS n FROM ${table} WHERE policy_id=?`).get(saved.id)).toEqual({ n: 1 });
|
||||
scaling.deleteScaling(saved.id, saved.revision);
|
||||
expect(scaling.getScalingPolicy(saved.id)).toBeUndefined();
|
||||
expect(scaling.listScalingPolicies().map(p => p.id)).toEqual([other.id]);
|
||||
for (const table of ['autoscaling_decisions', 'autoscaling_members', 'autoscaling_samples', 'autoscaling_authorizations']) expect(db().prepare(`SELECT COUNT(*) AS n FROM ${table} WHERE policy_id=?`).get(saved.id)).toEqual({ n: 0 });
|
||||
expect(db().prepare('SELECT name FROM servers').all()).toEqual([{ name: 't6_zombie_town_1' }]);
|
||||
expect(db().prepare('SELECT COUNT(*) AS n FROM server_keys').get()).toEqual({ n: 2 });
|
||||
expect(db().prepare('PRAGMA foreign_key_check').all()).toEqual([]);
|
||||
expect(scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token).id).not.toBe(saved.id);
|
||||
});
|
||||
|
||||
it.each(['off', 'create-only', 'create-destroy'])('deletes a settled %s policy even after key loss and releases ownership', async mode => {
|
||||
const { db, scaling, saved } = await setup(mode);
|
||||
db().prepare('DELETE FROM server_keys').run();
|
||||
scaling.deleteScaling(saved.id, saved.revision);
|
||||
const { scalingOwner } = await import('../../src/lib/server/autoscaling/ownership.js');
|
||||
expect(scalingOwner('t6', 'town')).toBeNull();
|
||||
expect(await scaling.runAutoscaling()).toBe(false);
|
||||
expect(db().prepare('SELECT COUNT(*) AS n FROM server_deployments').get()).toEqual({ n: 0 });
|
||||
});
|
||||
|
||||
it('rejects stale revisions and preserves the latest settings', async () => {
|
||||
const { scaling, saved } = await setup();
|
||||
const off = { ...settings, mode: 'off' };
|
||||
const edited = scaling.saveScaling(off, saved.id, saved.revision, scaling.previewScaling(off, saved.id, saved.revision).token);
|
||||
expect(() => scaling.deleteScaling(saved.id, saved.revision)).toThrow('changed');
|
||||
expect(scaling.getScalingPolicy(saved.id)).toEqual(edited);
|
||||
expect(() => scaling.deleteScaling('missing', 1)).toThrow('not found');
|
||||
});
|
||||
|
||||
it('blocks deletion with pending intent, then keeps Provisioning history after pause and settlement', async () => {
|
||||
const { db, scaling, saved } = await setup('create-only');
|
||||
await scaling.runAutoscaling();
|
||||
const { operation_id: id } = db().prepare('SELECT operation_id FROM autoscaling_decisions WHERE resolved=0').get() as { operation_id: string };
|
||||
expect(() => scaling.deleteScaling(saved.id, saved.revision)).toThrow('outstanding Provisioning');
|
||||
expect(scaling.getScalingPolicy(saved.id)?.mode).toBe('create-only');
|
||||
const off = { ...settings, mode: 'off' };
|
||||
const paused = scaling.saveScaling(off, saved.id, saved.revision, scaling.previewScaling(off, saved.id, saved.revision).token);
|
||||
scaling.deleteScaling(saved.id, paused.revision);
|
||||
const { getDeployment } = await import('../../src/lib/server/provisioning/history.js');
|
||||
expect(getDeployment(id)).toMatchObject({ deployment: { state: 'cancelled' }, sources: [expect.objectContaining({ kind: 'autoscaling', source_id: saved.id, label: 'Deleted autoscaling policy', href: null })] });
|
||||
});
|
||||
|
||||
it('validates delete requests and returns conflict, not-found, and success statuses', async () => {
|
||||
const { scaling, saved } = await setup();
|
||||
const { POST } = await import('../../src/routes/api/autoscaling/+server.js');
|
||||
const request = (body: object) => POST({ request: new Request('http://localhost/api/autoscaling', { method: 'POST', body: JSON.stringify({ action: 'delete', ...body }) }) });
|
||||
for (const body of [{}, { id: saved.id }, { id: saved.id, revision: '1' }, { id: saved.id, revision: 0 }]) expect((await request(body)).status).toBe(400);
|
||||
expect((await request({ id: saved.id, revision: saved.revision + 1 })).status).toBe(409);
|
||||
expect((await request({ id: saved.id, revision: saved.revision })).status).toBe(200);
|
||||
expect(scaling.getScalingPolicy(saved.id)).toBeUndefined();
|
||||
expect((await request({ id: saved.id, revision: saved.revision })).status).toBe(404);
|
||||
});
|
||||
@@ -3,14 +3,17 @@ import { newFixture, type Fixture } from '../helpers/fixture.js';
|
||||
import { routePermission, roleAllows } from '../../src/lib/server/auth/access.js';
|
||||
|
||||
let fixture: Fixture;
|
||||
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
||||
beforeEach(async () => {
|
||||
vi.resetModules(); fixture = await newFixture([]);
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
for (const n of [3, 8, 9, 10]) db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(`t6_zombie_town_${n}`, `secret-${n}`);
|
||||
});
|
||||
afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); });
|
||||
const settings = { game: 't6', mapSlug: 'town', maximum: 4, mode: 'dry-run' };
|
||||
|
||||
it('persists deduplicated dry-run decisions across restart without provisioning or reservations', async () => {
|
||||
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||
db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run('t6_zombie_town_3', 'secret');
|
||||
const preview = scaling.previewScaling(settings);
|
||||
const saved = scaling.saveScaling(settings, undefined, undefined, preview.token);
|
||||
await scaling.runAutoscaling(); await scaling.runAutoscaling();
|
||||
@@ -31,6 +34,8 @@ it('rejects duplicate policies, stale edits and changed previews', async () => {
|
||||
expect(() => scaling.saveScaling(settings, undefined, undefined, preview.token)).toThrow('already exists');
|
||||
expect(() => scaling.previewScaling(settings, saved.id, 0)).toThrow('changed');
|
||||
expect(() => scaling.saveScaling({ ...settings, maximum: 5 }, saved.id, saved.revision, preview.token)).toThrow('preview changed');
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
db().prepare("DELETE FROM server_keys WHERE server_id='t6_zombie_town_10'").run();
|
||||
expect(() => scaling.saveScaling({ ...settings, mode: 'create-destroy' }, saved.id, saved.revision, scaling.previewScaling({ ...settings, mode: 'create-destroy' }, saved.id, saved.revision).token)).toThrow('Save keys');
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
import { afterEach, beforeEach, expect, it, vi } from 'vitest';
|
||||
import { newFixture, type Fixture } from '../helpers/fixture.js';
|
||||
import { prepareNativeHost } from '../helpers/native-host.js';
|
||||
import { zombieMaps } from '../../src/lib/map-aliases.js';
|
||||
import { scalingSettings, type ScalingMember, type ScalingPolicy } from '../../src/lib/autoscaling.js';
|
||||
import { planScaling } from '../../src/lib/server/autoscaling/planner.js';
|
||||
|
||||
let fixture: Fixture;
|
||||
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
||||
afterEach(async () => { await fixture.cleanup(); });
|
||||
const settings = { game: 't6', mapSlug: '*', maximum: 2, mode: 'create-only' };
|
||||
const maps = zombieMaps('t6');
|
||||
async function setup() {
|
||||
await prepareNativeHost();
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||
for (const map of maps) for (const n of [1, 2]) db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(`t6_zombie_${map.slug}_${n}`, `key-${map.slug}-${n}`);
|
||||
return { db, scaling };
|
||||
}
|
||||
|
||||
it('applies targets and spare deficits independently to every map', () => {
|
||||
const now = 2_000_000;
|
||||
const policy: ScalingPolicy = { ...scalingSettings(settings), id: 'policy', revision: 1, nextActionAt: 0, pausedReason: null, updatedAt: '' };
|
||||
const members: ScalingMember[] = maps.flatMap(map => [1, 2].map(n => ({ name: `t6_zombie_${map.slug}_${n}`, mapSlug: map.slug,
|
||||
identity: `${map.slug}:${n}`, state: map.slug === 'town' ? 'occupied' : 'empty', players: map.slug === 'town' ? 1 : 0,
|
||||
observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null })));
|
||||
const snapshot = { members, reservedNames: members.map(m => m.name), keyedNames: members.map(m => m.name), blockers: [] };
|
||||
// Other maps' empty lobbies cannot satisfy Town's spare target or maximum.
|
||||
expect(planScaling(policy, snapshot, now)).toMatchObject({ action: 'destroy', server: 't6_zombie_transit_1', desired: maps.length + 1, deficit: 1 });
|
||||
snapshot.members = members.filter(m => m.name !== 't6_zombie_town_2');
|
||||
snapshot.reservedNames = snapshot.members.map(m => m.name);
|
||||
expect(planScaling(policy, snapshot, now)).toMatchObject({ action: 'create', server: 't6_zombie_town_2' });
|
||||
expect(planScaling({ ...policy, nextActionAt: now + 1 }, snapshot, now).action).toBe('wait');
|
||||
snapshot.members[0] = { ...snapshot.members[0], state: 'creating' };
|
||||
expect(planScaling(policy, snapshot, now).action).toBe('wait');
|
||||
});
|
||||
|
||||
it('persists a game rule, queues the selected map recipe, and protects every map', async () => {
|
||||
const { db, scaling } = await setup();
|
||||
db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run('t5_zombie_theater_1', 'other-game');
|
||||
const preview = scaling.previewScaling(settings);
|
||||
expect(preview.snapshot.keyedNames).toHaveLength(maps.length * 2);
|
||||
const saved = scaling.saveScaling(settings, undefined, undefined, preview.token);
|
||||
expect(() => scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token)).toThrow();
|
||||
expect(() => scaling.previewScaling({ ...settings, mapSlug: 'town' }, saved.id, saved.revision)).toThrow('cannot change');
|
||||
expect(await scaling.runAutoscaling()).toBe(true);
|
||||
const job = db().prepare('SELECT deployment_id,map_slug,server FROM server_creation_jobs').get() as { deployment_id: string; map_slug: string; server: string };
|
||||
expect(job).toMatchObject({ map_slug: 'transit', server: 't6_zombie_transit_1' });
|
||||
const { scalingAdmission } = await import('../../src/lib/server/autoscaling/admission.js');
|
||||
expect(scalingAdmission(job.deployment_id)).toBeNull();
|
||||
await scaling.runAutoscaling();
|
||||
expect(db().prepare('SELECT COUNT(*) AS n FROM server_creation_jobs').get()).toEqual({ n: 1 });
|
||||
const { scalingOwner, scalingOwnershipBlocker } = await import('../../src/lib/server/autoscaling/ownership.js');
|
||||
for (const map of maps) expect(scalingOwner('t6', map.slug)).toBe(saved.id);
|
||||
expect(scalingOwner('t5', 'theater')).toBeNull();
|
||||
expect(scalingOwnershipBlocker('t6_zombie_farm_1')).toContain(saved.id);
|
||||
expect(scaling.previewScaling({ ...settings, mapSlug: 'town' }).blockers.join(' ')).toContain('overlapping');
|
||||
const off = { ...settings, mode: 'off' };
|
||||
scaling.saveScaling(off, saved.id, saved.revision, scaling.previewScaling(off, saved.id, saved.revision).token);
|
||||
expect(scalingOwner('t6', 'town')).toBeNull();
|
||||
const mapSettings = { ...settings, mapSlug: 'town' };
|
||||
const mapPolicy = scaling.saveScaling(mapSettings, undefined, undefined, scaling.previewScaling(mapSettings).token);
|
||||
const current = scaling.getScalingPolicy(saved.id)!;
|
||||
expect(scaling.previewScaling(settings, saved.id, current.revision).blockers.join(' ')).toContain(mapPolicy.id);
|
||||
});
|
||||
|
||||
it('requires live key coverage on each map even when the game has enough keys in total', async () => {
|
||||
const { db, scaling } = await setup();
|
||||
db().prepare("DELETE FROM server_keys WHERE server_id='t6_zombie_town_2'").run();
|
||||
const preview = scaling.previewScaling(settings);
|
||||
expect(preview.blockers.join(' ')).toContain('Town Survival');
|
||||
expect(() => scaling.saveScaling(settings, undefined, undefined, preview.token)).toThrow('Town Survival');
|
||||
});
|
||||
@@ -0,0 +1,82 @@
|
||||
import { afterEach, beforeEach, expect, it, vi } from 'vitest';
|
||||
import { newFixture, type Fixture } from '../helpers/fixture.js';
|
||||
import { prepareNativeHost } from '../helpers/native-host.js';
|
||||
import { zombieMaps } from '../../src/lib/map-aliases.js';
|
||||
|
||||
let fixture: Fixture;
|
||||
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
||||
afterEach(async () => { await fixture.cleanup(); });
|
||||
const settings = { game: 't6', mapSlug: 'town', maximum: 2, mode: 'dry-run' };
|
||||
async function setup() {
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||
const key = (name: string, value = `key-${name}`) => db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(name, value);
|
||||
return { db, scaling, key };
|
||||
}
|
||||
|
||||
it.each(['off', 'dry-run', 'create-only', 'create-destroy'])('rejects new %s policies without enough keys through both service and API', async mode => {
|
||||
const { db, scaling, key } = await setup();
|
||||
key('t6_zombie_town_3');
|
||||
const input = { ...settings, mode };
|
||||
const preview = scaling.previewScaling(input);
|
||||
expect(preview.blockers[0]).toContain('1 saved; 1 more required');
|
||||
expect(() => scaling.saveScaling(input, undefined, undefined, preview.token)).toThrow('Save keys');
|
||||
const { POST } = await import('../../src/routes/api/autoscaling/+server.js');
|
||||
const response = await POST({ request: new Request('http://localhost/api/autoscaling', { method: 'POST', body: JSON.stringify({ action: 'save', settings: input, token: preview.token }) }) });
|
||||
expect(response.status).toBe(400);
|
||||
expect(await response.json()).toMatchObject({ error: expect.stringContaining('Save keys') });
|
||||
expect(db().prepare('SELECT COUNT(*) AS n FROM autoscaling_policies').get()).toEqual({ n: 0 });
|
||||
});
|
||||
|
||||
it('counts only supported, nonblank identities in the selected map and accepts exact nonconsecutive coverage', async () => {
|
||||
const { scaling, key } = await setup();
|
||||
for (const name of ['t6_zombie_town_0', 't6_zombie_town_21', 't6_zombie_town_01', 't6_zombie_town', 't6_town_2', 't5_zombie_theater_1', 't6_zombie_farm_1']) key(name);
|
||||
key('t6_zombie_town_2', ' ');
|
||||
key('t6_zombie_town_3');
|
||||
expect(scaling.previewScaling(settings).snapshot.keyedNames).toEqual(['t6_zombie_town_3']);
|
||||
key('t6_zombie_town_8');
|
||||
const preview = scaling.previewScaling(settings);
|
||||
expect(preview.blockers).toEqual([]);
|
||||
expect(scaling.saveScaling(settings, undefined, undefined, preview.token).maximum).toBe(2);
|
||||
});
|
||||
|
||||
it('requires coverage on every game map rather than pooling excess keys from another map', async () => {
|
||||
const { scaling, key } = await setup();
|
||||
for (const map of zombieMaps('t6')) for (const n of [3, 8]) if (map.slug !== 'town' || n === 3) key(`t6_zombie_${map.slug}_${n}`);
|
||||
key('t6_zombie_farm_9');
|
||||
const input = { ...settings, mapSlug: '*' };
|
||||
const preview = scaling.previewScaling(input);
|
||||
expect(preview.blockers).toEqual([expect.stringContaining('Town Survival')]);
|
||||
expect(() => scaling.saveScaling(input, undefined, undefined, preview.token)).toThrow('Town Survival');
|
||||
key('t6_zombie_town_8');
|
||||
expect(scaling.saveScaling(input, undefined, undefined, scaling.previewScaling(input).token).mapSlug).toBe('*');
|
||||
});
|
||||
|
||||
it('rechecks key deletion after preview and rejects increases beyond coverage', async () => {
|
||||
const { db, scaling, key } = await setup();
|
||||
key('t6_zombie_town_3'); key('t6_zombie_town_8');
|
||||
const preview = scaling.previewScaling(settings);
|
||||
db().prepare("DELETE FROM server_keys WHERE server_id='t6_zombie_town_8'").run();
|
||||
expect(() => scaling.saveScaling(settings, undefined, undefined, preview.token)).toThrow('preview changed');
|
||||
expect(() => scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token)).toThrow('Save keys');
|
||||
key('t6_zombie_town_8');
|
||||
const saved = scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token);
|
||||
const higher = { ...settings, maximum: 3 };
|
||||
expect(() => scaling.saveScaling(higher, saved.id, saved.revision, scaling.previewScaling(higher, saved.id, saved.revision).token)).toThrow('Save keys');
|
||||
expect(scaling.getScalingPolicy(saved.id)?.maximum).toBe(2);
|
||||
});
|
||||
|
||||
it('allows an existing live policy to pause and withdraw pending work after all keys are removed', async () => {
|
||||
await prepareNativeHost();
|
||||
const { db, scaling, key } = await setup();
|
||||
key('t6_zombie_town_3'); key('t6_zombie_town_8');
|
||||
const live = { ...settings, mode: 'create-only' };
|
||||
const saved = scaling.saveScaling(live, undefined, undefined, scaling.previewScaling(live).token);
|
||||
await scaling.runAutoscaling();
|
||||
db().prepare('DELETE FROM server_keys').run();
|
||||
const off = { ...settings, mode: 'off' };
|
||||
const preview = scaling.previewScaling(off, saved.id, saved.revision);
|
||||
expect(preview.blockers).toEqual([]);
|
||||
expect(scaling.saveScaling(off, saved.id, saved.revision, preview.token).mode).toBe('off');
|
||||
expect(db().prepare("SELECT state FROM server_deployments WHERE action='create'").get()).toEqual({ state: 'cancelled' });
|
||||
});
|
||||
@@ -1,5 +1,6 @@
|
||||
import { afterEach, beforeEach, expect, it, vi } from 'vitest';
|
||||
import { newFixture, type Fixture } from '../helpers/fixture.js';
|
||||
import { zombieMaps } from '../../src/lib/map-aliases.js';
|
||||
import { prepareNativeHost } from '../helpers/native-host.js';
|
||||
|
||||
vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconGetPlayerCount: vi.fn(async () => 0) }));
|
||||
@@ -9,24 +10,30 @@ afterEach(async () => { vi.useRealTimers(); await fixture.cleanup(); vi.restoreA
|
||||
const names = ['t6_zombie_town_1', 't6_zombie_town_2'];
|
||||
const settings = { game: 't6', mapSlug: 'town', maximum: 2, mode: 'create-destroy', emptyMinutes: 1, cooldownMinutes: 1 };
|
||||
|
||||
async function setup() {
|
||||
const host = await prepareNativeHost(names);
|
||||
async function setup(gameWide = false) {
|
||||
const servers = gameWide ? [...names, ...zombieMaps('t6').filter(m => m.slug !== 'town').map(m => `t6_zombie_${m.slug}_1`)] : names;
|
||||
const policySettings = { ...settings, mapSlug: gameWide ? '*' : settings.mapSlug };
|
||||
const host = await prepareNativeHost(servers);
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||
const authorization = await import('../../src/lib/server/autoscaling/authorization.js');
|
||||
db().prepare("UPDATE ec_hosts SET inventory=json_set(inventory,'$.capabilities',json(?)) WHERE id=?").run(JSON.stringify([authorization.autoscalingRemovalCapability]), host.host.id);
|
||||
const base = Date.now(); vi.useFakeTimers({ toFake: ['Date'] }); vi.setSystemTime(base);
|
||||
for (const name of names) {
|
||||
db().prepare("INSERT INTO servers(name,game,map,map_slug,container_state,last_players) VALUES(?,'t6','zm_transit','town','running',0)").run(name);
|
||||
for (const name of servers) {
|
||||
const map = zombieMaps('t6').find(m => name === `t6_zombie_${m.slug}_1` || name === `t6_zombie_${m.slug}_2`)!;
|
||||
db().prepare("INSERT INTO servers(name,game,map,map_slug,container_state,last_players) VALUES(?,'t6',?,?,'running',0)").run(name, map.canonical, map.slug);
|
||||
db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(name, `key-${name}`);
|
||||
}
|
||||
if (gameWide) for (const map of zombieMaps('t6').filter(m => m.slug !== 'town')) {
|
||||
db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(`t6_zombie_${map.slug}_2`, `key-${map.slug}-2`);
|
||||
}
|
||||
const fresh = (offset: number) => {
|
||||
vi.setSystemTime(base + offset); const at = new Date().toISOString();
|
||||
db().prepare('UPDATE servers SET container_checked_at=?,last_seen_at=?,last_checked_at=?').run(at, at, at);
|
||||
db().prepare('UPDATE ec_hosts SET last_heartbeat=?').run(at);
|
||||
};
|
||||
fresh(0);
|
||||
const saved = scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token);
|
||||
const saved = scaling.saveScaling(policySettings, undefined, undefined, scaling.previewScaling(policySettings).token);
|
||||
for (const offset of [0, 30_000, 60_000]) { fresh(offset); await scaling.runAutoscaling(); }
|
||||
const intent = db().prepare('SELECT operation_id FROM autoscaling_decisions WHERE resolved=0').get() as { operation_id: string };
|
||||
expect(intent?.operation_id).toBeTruthy();
|
||||
@@ -82,6 +89,8 @@ it('revokes a delivered removal on policy pause without claiming it was undone',
|
||||
expect(s.db().prepare('SELECT owns_map FROM autoscaling_policies').get()).toEqual({ owns_map: 1 });
|
||||
expect(s.db().prepare('SELECT state FROM server_deployments WHERE id=?').get(s.operationId)).toEqual({ state: 'running' });
|
||||
expect(s.scaling.scalingReport().policies[0]).toMatchObject({ ownsMap: true, unsettled: [{ id: s.operationId, state: 'running' }] });
|
||||
expect(() => s.scaling.deleteScaling(s.policy.id, s.scaling.getScalingPolicy(s.policy.id)!.revision)).toThrow('outstanding Provisioning');
|
||||
expect(s.db().prepare('SELECT COUNT(*) AS n FROM autoscaling_authorizations WHERE policy_id=?').get(s.policy.id)).toEqual({ n: 1 });
|
||||
});
|
||||
|
||||
it('withdraws undelivered commands and reconciles invalidation without a retry or failure pause', async () => {
|
||||
@@ -103,3 +112,10 @@ it('rejects expired, changed revision, and newly protected removal authorization
|
||||
s.fresh(130_000);
|
||||
expect(s.authorization.claimRemovalAuthorization(s.hostId, command.id, command.payload).authorized).toBe(false);
|
||||
});
|
||||
|
||||
it('authorizes a game-wide removal without trimming any map below its targets', async () => {
|
||||
const s = await setup(true);
|
||||
expect(s.db().prepare("SELECT server FROM server_deployments WHERE id=?").get(s.operationId)).toEqual({ server: names[0] });
|
||||
const command = await s.dispatch();
|
||||
expect(s.authorization.claimRemovalAuthorization(s.hostId, command.id, command.payload).authorized).toBe(true);
|
||||
});
|
||||
|
||||
@@ -20,6 +20,7 @@ it('separates variants, deduplicates pending identities, and initializes additiv
|
||||
expect(scalingSnapshot(policy).members.map(m => [m.name, m.state])).toEqual([[town, 'empty']]);
|
||||
db().prepare(`INSERT INTO server_deployments(id,server,action,state,stage,started_at,triggered_by) VALUES('create',?,'create','queued','Awaiting placement',?,'test')`).run(town, at);
|
||||
db().prepare(`INSERT INTO server_creation_jobs(deployment_id,server,game_mode,map_slug,expected_map) VALUES('create',?,'zombie','town','zm_transit')`).run(town);
|
||||
expect(scalingSnapshot({ ...policy, mapSlug: '*' }).members.map(m => m.name).sort()).toEqual([farm, town]);
|
||||
const snapshot = scalingSnapshot(policy);
|
||||
expect(snapshot.members).toHaveLength(1);
|
||||
expect(snapshot.members[0].state).toBe('creating');
|
||||
|
||||
Reference in New Issue
Block a user