Remove autoscaling action cooldown
This commit is contained in:
@@ -8,7 +8,7 @@ T5 game-wide policies cover the ten standard Zombies maps. Dead Ops Arcade is no
|
|||||||
|
|
||||||
## Configure a policy
|
## Configure a policy
|
||||||
|
|
||||||
Choose **Add policy**, select the game and scope, and enter an explicit maximum. The initial editor selects Off, with a minimum of one running lobby, one empty spare, a 20-minute empty interval, and a five-minute action cooldown. Counts must be between 1 and 20; the minimum and spare target cannot exceed the maximum.
|
Choose **Add policy**, select the game and scope, and enter an explicit maximum. The initial editor selects Off, with a minimum of one running lobby, one empty spare, and a 20-minute empty interval. Counts must be between 1 and 20; the minimum and spare target cannot exceed the maximum.
|
||||||
|
|
||||||
Choose **Preview changes** to inspect the existing lobbies, game key availability, target, proposed action, and blockers. **Save reviewed policy** checks the policy revision and current fleet again. If the preview changed, review a new preview before saving.
|
Choose **Preview changes** to inspect the existing lobbies, game key availability, target, proposed action, and blockers. **Save reviewed policy** checks the policy revision and current fleet again. If the preview changed, review a new preview before saving.
|
||||||
|
|
||||||
@@ -19,7 +19,7 @@ Choose **Preview changes** to inspect the existing lobbies, game key availabilit
|
|||||||
|
|
||||||
Plutonium keys belong to a game pool shared by every map, manual launch, collection, beta, and autoscaling policy for that game. An eligible creation uses the lowest available lobby number from 1–20 and reserves any free game key atomically. Key labels do not select a map or lobby. The preview shows saved and available game keys; policy maxima can exceed pool size, and a policy can be saved before any keys are added. An empty or exhausted pool holds new growth with a clear reason; it does not prevent surplus removal. Queued launches reserve their key, stopped instances retain it, and confirmed cleanup returns it to the pool. Failed or uncertain launches keep their reservation until safely reconciled. Existing lobbies and unfinished operations must be healthy and settled before activation. Automatic removal also requires agent **1.7.0 or later** advertising `instance.autoscaling-remove` on each affected host.
|
Plutonium keys belong to a game pool shared by every map, manual launch, collection, beta, and autoscaling policy for that game. An eligible creation uses the lowest available lobby number from 1–20 and reserves any free game key atomically. Key labels do not select a map or lobby. The preview shows saved and available game keys; policy maxima can exceed pool size, and a policy can be saved before any keys are added. An empty or exhausted pool holds new growth with a clear reason; it does not prevent surplus removal. Queued launches reserve their key, stopped instances retain it, and confirmed cleanup returns it to the pool. Failed or uncertain launches keep their reservation until safely reconciled. Existing lobbies and unfinished operations must be healthy and settled before activation. Automatic removal also requires agent **1.7.0 or later** advertising `instance.autoscaling-remove` on each affected host.
|
||||||
|
|
||||||
The desired count is `min(maximum, max(minimum, occupied lobbies + empty spare target))`. The controller evaluates policies every 15 seconds and requests at most one unresolved operation per policy. Game-wide policies share one cooldown and one outstanding operation across their maps, prioritizing additions before removals. Create uses the existing startup concurrency, CPU, memory, library, port, and host admission checks. Lack of host capacity leaves one request waiting; it does not create additional requests every tick. Creates use current promoted launch files when provisioning begins.
|
The desired count is `min(maximum, max(minimum, occupied lobbies + empty spare target))`. The controller evaluates policies every 15 seconds and requests at most one unresolved operation per policy. Game-wide policies allow one outstanding operation across their maps, prioritizing additions before removals. Once an operation settles, the next evaluation can act immediately on another eligible lobby. Surplus lobbies must still satisfy the configured continuous empty duration before removal. Create uses the existing startup concurrency, CPU, memory, library, port, and host admission checks. Lack of host capacity leaves one request waiting; it does not create additional requests every tick. Creates use current promoted launch files when provisioning begins.
|
||||||
|
|
||||||
Existing policies saved in the removed Dry run or Create only modes switch to Off on upgrade. Review and enable Create and destroy to resume; pending work remains tracked until settled.
|
Existing policies saved in the removed Dry run or Create only modes switch to Off on upgrade. Review and enable Create and destroy to resume; pending work remains tracked until settled.
|
||||||
|
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ Introduce an opt-in policy for each game and map variant that keeps a small numb
|
|||||||
|
|
||||||
Start with Create/Destroy on existing native hosts. Run the evaluator in dry run for two weeks, then enable creation for a small pilot before enabling removal. Host acquisition, VM shutdown, Stop/Start pooling, scheduled demand prediction, and waking servers from incoming packets are outside the first release.
|
Start with Create/Destroy on existing native hosts. Run the evaluator in dry run for two weeks, then enable creation for a small pilot before enabling removal. Host acquisition, VM shutdown, Stop/Start pooling, scheduled demand prediction, and waking servers from incoming packets are outside the first release.
|
||||||
|
|
||||||
Update October 7, 2026: Dry run and Create only have been removed. Current policies support Off and Create and destroy, defaulting to Off. Existing policies in removed modes switch to Off on upgrade. The original implementation stages below record the initial design; follow the current [usage and pilot procedure](../guides/autoscaling.md) for operation.
|
Update October 7, 2026: Dry run and Create only have been removed. Current policies support Off and Create and destroy, defaulting to Off. Existing policies in removed modes switch to Off on upgrade. Action cooldowns have also been removed; policies retain the continuous empty duration before removal and one outstanding operation at a time. The original implementation stages below record the initial design; follow the current [usage and pilot procedure](../guides/autoscaling.md) for operation.
|
||||||
|
|
||||||
## Existing foundations and required extensions
|
## Existing foundations and required extensions
|
||||||
|
|
||||||
|
|||||||
@@ -7,10 +7,10 @@ export interface ScalingSettings {
|
|||||||
/** '*' applies the targets independently to every map. */
|
/** '*' applies the targets independently to every map. */
|
||||||
mapSlug: string;
|
mapSlug: string;
|
||||||
mode: AutoscalingMode;
|
mode: AutoscalingMode;
|
||||||
minimum: number; maximum: number; spares: number; emptyMinutes: number; cooldownMinutes: number;
|
minimum: number; maximum: number; spares: number; emptyMinutes: number;
|
||||||
}
|
}
|
||||||
export interface ScalingPolicy extends ScalingSettings {
|
export interface ScalingPolicy extends ScalingSettings {
|
||||||
id: string; revision: number; nextActionAt: number; pausedReason: string | null; updatedAt: string;
|
id: string; revision: number; pausedReason: string | null; updatedAt: string;
|
||||||
}
|
}
|
||||||
export interface ScalingMember {
|
export interface ScalingMember {
|
||||||
name: string; mapSlug?: string; identity: string; state: 'occupied' | 'empty' | 'creating' | 'removing' | 'stopped' | 'unknown' | 'protected';
|
name: string; mapSlug?: string; identity: string; state: 'occupied' | 'empty' | 'creating' | 'removing' | 'stopped' | 'unknown' | 'protected';
|
||||||
@@ -52,5 +52,5 @@ export function scalingSettings(input: unknown): ScalingSettings {
|
|||||||
const minimum = integer('minimum', 1, 20), maximum = integer('maximum', undefined, 20), spares = integer('spares', 1, 20);
|
const minimum = integer('minimum', 1, 20), maximum = integer('maximum', undefined, 20), spares = integer('spares', 1, 20);
|
||||||
if (minimum > maximum || spares > maximum) throw new Error('Minimum and spare target cannot exceed the maximum.');
|
if (minimum > maximum || spares > maximum) throw new Error('Minimum and spare target cannot exceed the maximum.');
|
||||||
return { game, mapSlug, mode: mode as AutoscalingMode, minimum, maximum, spares,
|
return { game, mapSlug, mode: mode as AutoscalingMode, minimum, maximum, spares,
|
||||||
emptyMinutes: integer('emptyMinutes', 20, 1440), cooldownMinutes: integer('cooldownMinutes', 5, 1440) };
|
emptyMinutes: integer('emptyMinutes', 20, 1440) };
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -17,7 +17,7 @@ export function scalingAdmission(operationId: string): string | null {
|
|||||||
const paused = (db().prepare("SELECT value FROM settings WHERE key='autoscaling_paused'").get() as { value: string } | undefined)?.value === '1';
|
const paused = (db().prepare("SELECT value FROM settings WHERE key='autoscaling_paused'").get() as { value: string } | undefined)?.value === '1';
|
||||||
if (!row || !liveScaling(row.mode) || !row.owns_map || row.revision !== intent.revision || row.paused_reason || paused) reason = 'Autoscaling policy is paused or changed.';
|
if (!row || !liveScaling(row.mode) || !row.owns_map || row.revision !== intent.revision || row.paused_reason || paused) reason = 'Autoscaling policy is paused or changed.';
|
||||||
else {
|
else {
|
||||||
const policy: ScalingPolicy = { ...JSON.parse(row.settings), id: row.id, mode: row.mode, revision: row.revision, nextActionAt: 0, pausedReason: null, updatedAt: '' };
|
const policy: ScalingPolicy = { ...JSON.parse(row.settings), id: row.id, mode: row.mode, revision: row.revision, pausedReason: null, updatedAt: '' };
|
||||||
const requested = JSON.parse(intent.decision) as ScalingDecision;
|
const requested = JSON.parse(intent.decision) as ScalingDecision;
|
||||||
const snapshot = scalingSnapshot(policy, Date.now(), operationId);
|
const snapshot = scalingSnapshot(policy, Date.now(), operationId);
|
||||||
if (requested.action === 'create') {
|
if (requested.action === 'create') {
|
||||||
|
|||||||
@@ -15,8 +15,8 @@ import { scalingOwner } from './ownership.js';
|
|||||||
import { getServer } from '../fleet/servers.js';
|
import { getServer } from '../fleet/servers.js';
|
||||||
|
|
||||||
export class ScalingError extends Error { constructor(message: string, public status = 400) { super(message); } }
|
export class ScalingError extends Error { constructor(message: string, public status = 400) { super(message); } }
|
||||||
interface PolicyRow { id: string; revision: number; settings: string; mode: ScalingPolicy['mode']; next_action_at: number; paused_reason: string | null; updated_at: string }
|
interface PolicyRow { id: string; revision: number; settings: string; mode: ScalingPolicy['mode']; paused_reason: string | null; updated_at: string }
|
||||||
const present = (row: PolicyRow): ScalingPolicy => ({ ...JSON.parse(row.settings), id: row.id, revision: row.revision, mode: row.mode, nextActionAt: row.next_action_at, pausedReason: row.paused_reason, updatedAt: row.updated_at });
|
const present = (row: PolicyRow): ScalingPolicy => ({ ...JSON.parse(row.settings), id: row.id, revision: row.revision, mode: row.mode, pausedReason: row.paused_reason, updatedAt: row.updated_at });
|
||||||
export const listScalingPolicies = (): ScalingPolicy[] => (db().prepare('SELECT * FROM autoscaling_policies ORDER BY game,map_slug').all() as PolicyRow[]).map(present);
|
export const listScalingPolicies = (): ScalingPolicy[] => (db().prepare('SELECT * FROM autoscaling_policies ORDER BY game,map_slug').all() as PolicyRow[]).map(present);
|
||||||
export function getScalingPolicy(id: string): ScalingPolicy | undefined { const row = db().prepare('SELECT * FROM autoscaling_policies WHERE id=?').get(id) as PolicyRow | undefined; return row && present(row); }
|
export function getScalingPolicy(id: string): ScalingPolicy | undefined { const row = db().prepare('SELECT * FROM autoscaling_policies WHERE id=?').get(id) as PolicyRow | undefined; return row && present(row); }
|
||||||
export const scalingGloballyPaused = (): boolean => (db().prepare("SELECT value FROM settings WHERE key='autoscaling_paused'").get() as { value: string } | undefined)?.value === '1';
|
export const scalingGloballyPaused = (): boolean => (db().prepare("SELECT value FROM settings WHERE key='autoscaling_paused'").get() as { value: string } | undefined)?.value === '1';
|
||||||
@@ -27,7 +27,7 @@ export function previewScaling(input: unknown, id?: string, revision?: number):
|
|||||||
if (existing && existing.revision !== revision) throw new ScalingError('Policy changed. Refresh before editing.', 409);
|
if (existing && existing.revision !== revision) throw new ScalingError('Policy changed. Refresh before editing.', 409);
|
||||||
if (existing && (existing.game !== settings.game || existing.mapSlug !== settings.mapSlug)) throw new ScalingError('A policy cannot change its game or map.');
|
if (existing && (existing.game !== settings.game || existing.mapSlug !== settings.mapSlug)) throw new ScalingError('A policy cannot change its game or map.');
|
||||||
const policy: ScalingPolicy = { ...settings, id: existing?.id ?? 'preview', revision: existing?.revision ?? 0,
|
const policy: ScalingPolicy = { ...settings, id: existing?.id ?? 'preview', revision: existing?.revision ?? 0,
|
||||||
nextActionAt: existing?.nextActionAt ?? 0, pausedReason: null, updatedAt: existing?.updatedAt ?? '' };
|
pausedReason: null, updatedAt: existing?.updatedAt ?? '' };
|
||||||
const snapshot = scalingSnapshot(policy);
|
const snapshot = scalingSnapshot(policy);
|
||||||
const blockers: string[] = [];
|
const blockers: string[] = [];
|
||||||
if (liveScaling(policy.mode)) {
|
if (liveScaling(policy.mode)) {
|
||||||
@@ -167,7 +167,6 @@ export async function runAutoscaling(): Promise<boolean> {
|
|||||||
if (!result.deploymentId) throw new Error('Provisioning did not return an operation identity.');
|
if (!result.deploymentId) throw new Error('Provisioning did not return an operation identity.');
|
||||||
db().prepare('UPDATE autoscaling_decisions SET operation_id=? WHERE id=?').run(result.deploymentId, decisionId);
|
db().prepare('UPDATE autoscaling_decisions SET operation_id=? WHERE id=?').run(result.deploymentId, decisionId);
|
||||||
db().prepare("INSERT INTO provisioning_sources(operation_id,kind,source_id,server) VALUES(?,'autoscaling',?,?)").run(result.deploymentId, policy.id, decision.server);
|
db().prepare("INSERT INTO provisioning_sources(operation_id,kind,source_id,server) VALUES(?,'autoscaling',?,?)").run(result.deploymentId, policy.id, decision.server);
|
||||||
db().prepare('UPDATE autoscaling_policies SET next_action_at=? WHERE id=?').run(now + policy.cooldownMinutes * 60_000, policy.id);
|
|
||||||
submitted = true;
|
submitted = true;
|
||||||
})();
|
})();
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
|||||||
@@ -47,7 +47,6 @@ export function planScaling(policy: ScalingPolicy, snapshot: ScalingSnapshot, no
|
|||||||
const blocked = members.find(m => !['occupied', 'empty'].includes(m.state));
|
const blocked = members.find(m => !['occupied', 'empty'].includes(m.state));
|
||||||
if (blocked) return result('wait', blocked.blocker ?? `Waiting for ${blocked.name}: ${blocked.state}.`);
|
if (blocked) return result('wait', blocked.blocker ?? `Waiting for ${blocked.name}: ${blocked.state}.`);
|
||||||
if (members.some(m => m.observedAt === null || m.observedAt > now || now - m.observedAt > ONLINE_MAX_MS)) return result('wait', 'Waiting for fresh occupancy evidence.');
|
if (members.some(m => m.observedAt === null || m.observedAt > now || now - m.observedAt > ONLINE_MAX_MS)) return result('wait', 'Waiting for fresh occupancy evidence.');
|
||||||
if (policy.nextActionAt > now) return result('wait', 'Action cooldown is active.');
|
|
||||||
if (occupied + empty < desired) {
|
if (occupied + empty < desired) {
|
||||||
if (snapshot.reservedNames.length >= policy.maximum) return result('wait', 'Maximum lobby reservations reached.');
|
if (snapshot.reservedNames.length >= policy.maximum) return result('wait', 'Maximum lobby reservations reached.');
|
||||||
if (!snapshot.keyPool.available) return result('wait', snapshot.keyPool.total
|
if (!snapshot.keyPool.available) return result('wait', snapshot.keyPool.total
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ export function ensureAutoscalingSchema(handle: Database.Database): void {
|
|||||||
CREATE TABLE IF NOT EXISTS autoscaling_policies (
|
CREATE TABLE IF NOT EXISTS autoscaling_policies (
|
||||||
id TEXT PRIMARY KEY, game TEXT NOT NULL, map_slug TEXT NOT NULL,
|
id TEXT PRIMARY KEY, game TEXT NOT NULL, map_slug TEXT NOT NULL,
|
||||||
revision INTEGER NOT NULL DEFAULT 1, settings TEXT NOT NULL,
|
revision INTEGER NOT NULL DEFAULT 1, settings TEXT NOT NULL,
|
||||||
mode TEXT NOT NULL DEFAULT 'off', next_action_at INTEGER NOT NULL DEFAULT 0,
|
mode TEXT NOT NULL DEFAULT 'off',
|
||||||
paused_reason TEXT, owns_map INTEGER NOT NULL DEFAULT 0, updated_at TEXT NOT NULL,
|
paused_reason TEXT, owns_map INTEGER NOT NULL DEFAULT 0, updated_at TEXT NOT NULL,
|
||||||
UNIQUE(game,map_slug)
|
UNIQUE(game,map_slug)
|
||||||
);
|
);
|
||||||
@@ -34,9 +34,15 @@ export function ensureAutoscalingSchema(handle: Database.Database): void {
|
|||||||
);
|
);
|
||||||
CREATE INDEX IF NOT EXISTS autoscaling_sample_retention ON autoscaling_samples(sampled_at);
|
CREATE INDEX IF NOT EXISTS autoscaling_sample_retention ON autoscaling_samples(sampled_at);
|
||||||
`);
|
`);
|
||||||
// Removed modes must not silently gain permission to destroy lobbies.
|
|
||||||
// Keep ownership while delivered work settles through normal reconciliation.
|
|
||||||
handle.transaction(() => {
|
handle.transaction(() => {
|
||||||
|
const columns = handle.pragma('table_info(autoscaling_policies)') as { name: string }[];
|
||||||
|
if (columns.some(column => column.name === 'next_action_at')) {
|
||||||
|
handle.exec('ALTER TABLE autoscaling_policies DROP COLUMN next_action_at');
|
||||||
|
}
|
||||||
|
handle.exec(`UPDATE autoscaling_policies SET settings=json_remove(settings,'$.cooldownMinutes')
|
||||||
|
WHERE json_type(settings,'$.cooldownMinutes') IS NOT NULL`);
|
||||||
|
// Removed modes must not silently gain permission to destroy lobbies.
|
||||||
|
// Keep ownership while delivered work settles through normal reconciliation.
|
||||||
handle.prepare(`UPDATE autoscaling_authorizations SET revoked_at=?
|
handle.prepare(`UPDATE autoscaling_authorizations SET revoked_at=?
|
||||||
WHERE claimed_at IS NULL AND revoked_at IS NULL AND policy_id IN
|
WHERE claimed_at IS NULL AND revoked_at IS NULL AND policy_id IN
|
||||||
(SELECT id FROM autoscaling_policies WHERE mode IN ('dry-run','create-only'))`).run(Date.now());
|
(SELECT id FROM autoscaling_policies WHERE mode IN ('dry-run','create-only'))`).run(Date.now());
|
||||||
|
|||||||
@@ -12,7 +12,7 @@
|
|||||||
let snapshot = $state<ScalingReport | null>(null);
|
let snapshot = $state<ScalingReport | null>(null);
|
||||||
const report = $derived(snapshot ?? data.autoscaling);
|
const report = $derived(snapshot ?? data.autoscaling);
|
||||||
const canOperate = $derived(page.data.user?.role === 'admin' || page.data.user?.role === 'operator');
|
const canOperate = $derived(page.data.user?.role === 'admin' || page.data.user?.role === 'operator');
|
||||||
const defaults = (): ScalingSettings => ({ game: 't5', mapSlug: 'theater', mode: 'off', minimum: 1, maximum: 4, spares: 1, emptyMinutes: 20, cooldownMinutes: 5 });
|
const defaults = (): ScalingSettings => ({ game: 't5', mapSlug: 'theater', mode: 'off', minimum: 1, maximum: 4, spares: 1, emptyMinutes: 20 });
|
||||||
let settings = $state<ScalingSettings>(defaults());
|
let settings = $state<ScalingSettings>(defaults());
|
||||||
let editing = $state<ScalingPolicy | null>(null);
|
let editing = $state<ScalingPolicy | null>(null);
|
||||||
let editorOpen = $state(false);
|
let editorOpen = $state(false);
|
||||||
@@ -92,9 +92,8 @@
|
|||||||
<label>Maximum lobbies<input type="number" min="1" max="20" required bind:value={settings.maximum} /></label>
|
<label>Maximum lobbies<input type="number" min="1" max="20" required bind:value={settings.maximum} /></label>
|
||||||
<label>Empty spare target<input type="number" min="1" max="20" required bind:value={settings.spares} /></label>
|
<label>Empty spare target<input type="number" min="1" max="20" required bind:value={settings.spares} /></label>
|
||||||
<label>Empty duration in minutes<input type="number" min="1" max="1440" required bind:value={settings.emptyMinutes} /></label>
|
<label>Empty duration in minutes<input type="number" min="1" max="1440" required bind:value={settings.emptyMinutes} /></label>
|
||||||
<label>Action cooldown in minutes<input type="number" min="1" max="1440" required bind:value={settings.cooldownMinutes} /></label>
|
|
||||||
</div>
|
</div>
|
||||||
{#if settings.mapSlug === '*'}<p>Targets apply separately to every map, including all variants. All maps share this game’s key pool. Actions share one cooldown and run one at a time across the game.</p>{/if}
|
{#if settings.mapSlug === '*'}<p>Targets apply separately to every map, including all variants. All maps share this game’s key pool. Actions run one at a time across the game.</p>{/if}
|
||||||
<p>Each new lobby needs an available key from the game pool. Creation waits when the pool is exhausted; keys return after confirmed removal.</p>
|
<p>Each new lobby needs an available key from the game pool. Creation waits when the pool is exhausted; keys return after confirmed removal.</p>
|
||||||
<div class="actions"><Button type="submit" disabled={busy}>Preview changes</Button><Button onclick={() => editorOpen = false}>Close editor</Button></div>
|
<div class="actions"><Button type="submit" disabled={busy}>Preview changes</Button><Button onclick={() => editorOpen = false}>Close editor</Button></div>
|
||||||
</fieldset>
|
</fieldset>
|
||||||
@@ -120,7 +119,7 @@
|
|||||||
{#if view.unsettled.length}<div class="banner"><strong>Work is still settling.</strong><p>Pause this policy and settle its outstanding work before deleting it.</p>{#each view.unsettled as operation}<p><a href={`/provisioning/${operation.id}`}>{operation.action} · {operation.state}</a> — {operation.stage}</p>{/each}{#if view.ownsMap}<p>This policy retains ownership of its scope until its outstanding work settles.</p>{/if}</div>{/if}
|
{#if view.unsettled.length}<div class="banner"><strong>Work is still settling.</strong><p>Pause this policy and settle its outstanding work before deleting it.</p>{#each view.unsettled as operation}<p><a href={`/provisioning/${operation.id}`}>{operation.action} · {operation.state}</a> — {operation.stage}</p>{/each}{#if view.ownsMap}<p>This policy retains ownership of its scope until its outstanding work settles.</p>{/if}</div>{/if}
|
||||||
<p>{view.policy.mapSlug === '*' ? 'Per map: ' : ''}Minimum {view.policy.minimum} · Maximum {view.policy.maximum} · Spare target {view.policy.spares} · {view.snapshot.keyPool.available} / {view.snapshot.keyPool.total} game keys available</p>
|
<p>{view.policy.mapSlug === '*' ? 'Per map: ' : ''}Minimum {view.policy.minimum} · Maximum {view.policy.maximum} · Spare target {view.policy.spares} · {view.snapshot.keyPool.available} / {view.snapshot.keyPool.total} game keys available</p>
|
||||||
<p><strong>Current decision: </strong>{view.decision.action}{view.decision.server ? ` ${view.decision.server}` : ''}. {view.decision.reason}</p>
|
<p><strong>Current decision: </strong>{view.decision.action}{view.decision.server ? ` ${view.decision.server}` : ''}. {view.decision.reason}</p>
|
||||||
<p class="muted">Last evaluated: {view.lastEvaluatedAt ? $timeFormat.dateTime(new Date(view.lastEvaluatedAt).toISOString()) : 'Not evaluated yet'}{view.policy.nextActionAt > Date.now() ? ` · Next action after ${$timeFormat.dateTime(new Date(view.policy.nextActionAt).toISOString())}` : ''}</p>
|
<p class="muted">Last evaluated: {view.lastEvaluatedAt ? $timeFormat.dateTime(new Date(view.lastEvaluatedAt).toISOString()) : 'Not evaluated yet'}</p>
|
||||||
<details><summary>Lobbies and evidence</summary><ul>{#each view.snapshot.members as member}<li><a href={`/servers/${encodeURIComponent(member.name)}`}>{member.name}</a> · {member.state}{member.blocker ? ` · ${member.blocker}` : ''}<br />{member.source ?? 'No current source'} · {member.observedAt ? $timeFormat.dateTime(new Date(member.observedAt).toISOString()) : 'Unknown observation time'}{#if member.operationId} · <a href={`/provisioning/${member.operationId}`}>Provisioning</a>{/if}</li>{/each}</ul>{#if !view.snapshot.members.length}<p>No registered or pending lobbies.</p>{/if}{#each view.snapshot.blockers as blocker}<p>{blocker}</p>{/each}</details>
|
<details><summary>Lobbies and evidence</summary><ul>{#each view.snapshot.members as member}<li><a href={`/servers/${encodeURIComponent(member.name)}`}>{member.name}</a> · {member.state}{member.blocker ? ` · ${member.blocker}` : ''}<br />{member.source ?? 'No current source'} · {member.observedAt ? $timeFormat.dateTime(new Date(member.observedAt).toISOString()) : 'Unknown observation time'}{#if member.operationId} · <a href={`/provisioning/${member.operationId}`}>Provisioning</a>{/if}</li>{/each}</ul>{#if !view.snapshot.members.length}<p>No registered or pending lobbies.</p>{/if}{#each view.snapshot.blockers as blocker}<p>{blocker}</p>{/each}</details>
|
||||||
<details><summary>Recent decisions and measurements</summary>
|
<details><summary>Recent decisions and measurements</summary>
|
||||||
<p>{view.totals.observedMinutes.toFixed(1)} observed minutes · {view.totals.emptyLobbyHours.toFixed(2)} empty lobby-hours · {view.totals.deficitMinutes.toFixed(1)} minutes below spare target · {view.totals.unknownMinutes.toFixed(1)} unknown minutes</p>
|
<p>{view.totals.observedMinutes.toFixed(1)} observed minutes · {view.totals.emptyLobbyHours.toFixed(2)} empty lobby-hours · {view.totals.deficitMinutes.toFixed(1)} minutes below spare target · {view.totals.unknownMinutes.toFixed(1)} unknown minutes</p>
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ const maps = zombieMaps('t6');
|
|||||||
it('satisfies a T5 game-wide policy with the ten standard Zombies maps', () => {
|
it('satisfies a T5 game-wide policy with the ten standard Zombies maps', () => {
|
||||||
const now = 2_000_000;
|
const now = 2_000_000;
|
||||||
const policy: ScalingPolicy = { ...scalingSettings({ ...settings, game: 't5' }),
|
const policy: ScalingPolicy = { ...scalingSettings({ ...settings, game: 't5' }),
|
||||||
id: 'policy', revision: 1, nextActionAt: 0, pausedReason: null, updatedAt: '' };
|
id: 'policy', revision: 1, pausedReason: null, updatedAt: '' };
|
||||||
const members: ScalingMember[] = zombieMaps('t5').map(map => ({ name: `t5_zombie_${map.slug}_1`, mapSlug: map.slug,
|
const members: ScalingMember[] = zombieMaps('t5').map(map => ({ name: `t5_zombie_${map.slug}_1`, mapSlug: map.slug,
|
||||||
identity: map.slug, state: 'empty', players: 0, observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null }));
|
identity: map.slug, state: 'empty', players: 0, observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null }));
|
||||||
const snapshot = { members, reservedNames: members.map(m => m.name), keyPool: { total: 20, available: 10 }, blockers: [] };
|
const snapshot = { members, reservedNames: members.map(m => m.name), keyPool: { total: 20, available: 10 }, blockers: [] };
|
||||||
@@ -32,7 +32,7 @@ async function setup() {
|
|||||||
|
|
||||||
it('applies targets and spare deficits independently to every map', () => {
|
it('applies targets and spare deficits independently to every map', () => {
|
||||||
const now = 2_000_000;
|
const now = 2_000_000;
|
||||||
const policy: ScalingPolicy = { ...scalingSettings(settings), id: 'policy', revision: 1, nextActionAt: 0, pausedReason: null, updatedAt: '' };
|
const policy: ScalingPolicy = { ...scalingSettings(settings), id: 'policy', revision: 1, pausedReason: null, updatedAt: '' };
|
||||||
const members: ScalingMember[] = maps.flatMap(map => [1, 2].map(n => ({ name: `t6_zombie_${map.slug}_${n}`, mapSlug: map.slug,
|
const members: ScalingMember[] = maps.flatMap(map => [1, 2].map(n => ({ name: `t6_zombie_${map.slug}_${n}`, mapSlug: map.slug,
|
||||||
identity: `${map.slug}:${n}`, state: map.slug === 'town' ? 'occupied' : 'empty', players: map.slug === 'town' ? 1 : 0,
|
identity: `${map.slug}:${n}`, state: map.slug === 'town' ? 'occupied' : 'empty', players: map.slug === 'town' ? 1 : 0,
|
||||||
observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null })));
|
observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null })));
|
||||||
@@ -42,7 +42,6 @@ it('applies targets and spare deficits independently to every map', () => {
|
|||||||
snapshot.members = members.filter(m => m.name !== 't6_zombie_town_2');
|
snapshot.members = members.filter(m => m.name !== 't6_zombie_town_2');
|
||||||
snapshot.reservedNames = snapshot.members.map(m => m.name);
|
snapshot.reservedNames = snapshot.members.map(m => m.name);
|
||||||
expect(planScaling(policy, snapshot, now)).toMatchObject({ action: 'create', server: 't6_zombie_town_2' });
|
expect(planScaling(policy, snapshot, now)).toMatchObject({ action: 'create', server: 't6_zombie_town_2' });
|
||||||
expect(planScaling({ ...policy, nextActionAt: now + 1 }, snapshot, now).action).toBe('wait');
|
|
||||||
snapshot.members[0] = { ...snapshot.members[0], state: 'creating' };
|
snapshot.members[0] = { ...snapshot.members[0], state: 'creating' };
|
||||||
expect(planScaling(policy, snapshot, now).action).toBe('wait');
|
expect(planScaling(policy, snapshot, now).action).toBe('wait');
|
||||||
});
|
});
|
||||||
@@ -87,3 +86,33 @@ it('holds game-wide growth when its shared pool is exhausted by another map', as
|
|||||||
expect(await scaling.runAutoscaling()).toBe(false);
|
expect(await scaling.runAutoscaling()).toBe(false);
|
||||||
expect(db().prepare('SELECT COUNT(*) AS n FROM server_creation_jobs').get()).toEqual({ n: 0 });
|
expect(db().prepare('SELECT COUNT(*) AS n FROM server_creation_jobs').get()).toEqual({ n: 0 });
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('moves to the next map as soon as the prior creation settles, without waiting between actions', async () => {
|
||||||
|
const { host, versions } = await prepareNativeHost();
|
||||||
|
const { db } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const { config } = await import('../../src/lib/server/config.js');
|
||||||
|
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||||
|
for (const n of [1, 2]) db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${n}`, `game-key-${n}`);
|
||||||
|
vi.useFakeTimers({ toFake: ['Date'] });
|
||||||
|
try {
|
||||||
|
const at = new Date().toISOString();
|
||||||
|
scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token);
|
||||||
|
expect(await scaling.runAutoscaling()).toBe(true);
|
||||||
|
const job = db().prepare('SELECT deployment_id,server FROM server_creation_jobs').get() as { deployment_id: string; server: string };
|
||||||
|
expect(job.server).toBe('t6_zombie_transit_1');
|
||||||
|
expect(await scaling.runAutoscaling()).toBe(false);
|
||||||
|
expect(db().prepare('SELECT COUNT(*) AS n FROM server_creation_jobs').get()).toEqual({ n: 1 });
|
||||||
|
|
||||||
|
// Record the first lobby becoming ready without advancing the clock.
|
||||||
|
db().prepare("UPDATE server_deployments SET state='done',finished_at=? WHERE id=?").run(at, job.deployment_id);
|
||||||
|
db().prepare(`INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at)
|
||||||
|
VALUES('ready-instance',?,?,?,?,28960,?)`).run(job.server, host.id, versions.get('t6'), config.defaultRuntimeImage, at);
|
||||||
|
db().prepare(`INSERT INTO servers(name,game,map,map_slug,container_state,container_checked_at,last_seen_at,last_checked_at,last_players)
|
||||||
|
VALUES(?,'t6','zm_transit','transit','running',?,?,?,0)`).run(job.server, at, at, at);
|
||||||
|
expect(await scaling.runAutoscaling()).toBe(true);
|
||||||
|
expect(db().prepare('SELECT server FROM server_creation_jobs ORDER BY rowid').all()).toEqual([
|
||||||
|
{ server: 't6_zombie_transit_1' }, { server: 't6_zombie_town_1' }
|
||||||
|
]);
|
||||||
|
expect(db().prepare('SELECT COUNT(*) AS n FROM autoscaling_decisions WHERE resolved=0').get()).toEqual({ n: 1 });
|
||||||
|
} finally { vi.useRealTimers(); }
|
||||||
|
});
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ import { emptyObservation, planScaling } from '../../src/lib/server/autoscaling/
|
|||||||
import { mapIdentity } from '../../src/lib/server/autoscaling/snapshot.js';
|
import { mapIdentity } from '../../src/lib/server/autoscaling/snapshot.js';
|
||||||
|
|
||||||
const now = 2_000_000;
|
const now = 2_000_000;
|
||||||
const policy: ScalingPolicy = { ...scalingSettings({ game: 't6', mapSlug: 'town', maximum: 4, mode: 'create-destroy' }), id: 'policy', revision: 1, nextActionAt: 0, pausedReason: null, updatedAt: '' };
|
const policy: ScalingPolicy = { ...scalingSettings({ game: 't6', mapSlug: 'town', maximum: 4, mode: 'create-destroy' }), id: 'policy', revision: 1, pausedReason: null, updatedAt: '' };
|
||||||
const name = (n: number) => `t6_zombie_town_${n}`;
|
const name = (n: number) => `t6_zombie_town_${n}`;
|
||||||
const member = (n: number, state: ScalingMember['state'] = 'empty'): ScalingMember => ({ name: name(n), identity: `instance:${n}`, state, players: state === 'occupied' ? 1 : state === 'empty' ? 0 : null, observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null });
|
const member = (n: number, state: ScalingMember['state'] = 'empty'): ScalingMember => ({ name: name(n), identity: `instance:${n}`, state, players: state === 'occupied' ? 1 : state === 'empty' ? 0 : null, observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null });
|
||||||
const snapshot = (...members: ScalingMember[]): ScalingSnapshot => ({ members, reservedNames: members.map(m => m.name), keyPool: { total: 4, available: 4 - members.length }, blockers: [] });
|
const snapshot = (...members: ScalingMember[]): ScalingSnapshot => ({ members, reservedNames: members.map(m => m.name), keyPool: { total: 4, available: 4 - members.length }, blockers: [] });
|
||||||
@@ -31,9 +31,8 @@ describe('autoscaling policy planning', () => {
|
|||||||
for (const state of ['creating', 'removing', 'unknown', 'stopped', 'protected'] as const) expect(planScaling(policy, snapshot(member(1, state)), now).action).toBe('wait');
|
for (const state of ['creating', 'removing', 'unknown', 'stopped', 'protected'] as const) expect(planScaling(policy, snapshot(member(1, state)), now).action).toBe('wait');
|
||||||
expect(planScaling(policy, { ...snapshot(), blockers: ['Unresolved instance'] }, now).action).toBe('wait');
|
expect(planScaling(policy, { ...snapshot(), blockers: ['Unresolved instance'] }, now).action).toBe('wait');
|
||||||
});
|
});
|
||||||
it('holds for stale and future samples, cooldown, and disabled policies', () => {
|
it('holds for stale and future samples and disabled policies', () => {
|
||||||
for (const stamp of [null, now - 90_001, now + 1]) expect(planScaling(policy, snapshot({ ...member(1, 'occupied'), observedAt: stamp }), now).action).toBe('wait');
|
for (const stamp of [null, now - 90_001, now + 1]) expect(planScaling(policy, snapshot({ ...member(1, 'occupied'), observedAt: stamp }), now).action).toBe('wait');
|
||||||
expect(planScaling({ ...policy, nextActionAt: now + 1 }, snapshot(), now).action).toBe('wait');
|
|
||||||
expect(planScaling({ ...policy, mode: 'off' }, snapshot(), now).action).toBe('wait');
|
expect(planScaling({ ...policy, mode: 'off' }, snapshot(), now).action).toBe('wait');
|
||||||
});
|
});
|
||||||
it('never sacrifices occupied lobbies or minimum/spare capacity to a maximum', () => {
|
it('never sacrifices occupied lobbies or minimum/spare capacity to a maximum', () => {
|
||||||
|
|||||||
@@ -114,3 +114,25 @@ it.each([false, true])('reconciles removed-mode creation while preserving submit
|
|||||||
expect(db().prepare('SELECT COUNT(*) AS n FROM server_key_leases').get()).toEqual({ n: submitted ? 1 : 0 });
|
expect(db().prepare('SELECT COUNT(*) AS n FROM server_key_leases').get()).toEqual({ n: submitted ? 1 : 0 });
|
||||||
expect(db().prepare("SELECT COUNT(*) AS n FROM server_deployments WHERE action='create'").get()).toEqual({ n: 1 });
|
expect(db().prepare("SELECT COUNT(*) AS n FROM server_deployments WHERE action='create'").get()).toEqual({ n: 1 });
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('removes saved cooldowns on upgrade without changing the empty hold or pausing the policy', async () => {
|
||||||
|
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||||
|
const scaling = await import('../../src/lib/server/autoscaling/index.js');
|
||||||
|
const input = { ...settings, emptyMinutes: 17 };
|
||||||
|
const saved = scaling.saveScaling(input, undefined, undefined, scaling.previewScaling(input).token);
|
||||||
|
db().exec('ALTER TABLE autoscaling_policies ADD COLUMN next_action_at INTEGER NOT NULL DEFAULT 0');
|
||||||
|
db().prepare("UPDATE autoscaling_policies SET next_action_at=?,settings=json_set(settings,'$.cooldownMinutes',60) WHERE id=?")
|
||||||
|
.run(Date.now() + 3600_000, saved.id);
|
||||||
|
closeDb();
|
||||||
|
const restored = scaling.getScalingPolicy(saved.id)!;
|
||||||
|
expect(restored).toEqual(saved);
|
||||||
|
expect(restored).not.toHaveProperty('cooldownMinutes');
|
||||||
|
expect(restored).not.toHaveProperty('nextActionAt');
|
||||||
|
expect(db().pragma('table_info(autoscaling_policies)')).not.toContainEqual(expect.objectContaining({ name: 'next_action_at' }));
|
||||||
|
expect(JSON.parse((db().prepare('SELECT settings FROM autoscaling_policies WHERE id=?').get(saved.id) as { settings: string }).settings))
|
||||||
|
.not.toHaveProperty('cooldownMinutes');
|
||||||
|
closeDb();
|
||||||
|
expect(scaling.getScalingPolicy(saved.id)).toEqual(saved);
|
||||||
|
expect(await scaling.runAutoscaling()).toBe(true);
|
||||||
|
expect(scaling.scalingReport().policies[0].policy.emptyMinutes).toBe(17);
|
||||||
|
});
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ let fixture: Fixture;
|
|||||||
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
||||||
afterEach(async () => { vi.useRealTimers(); await fixture.cleanup(); vi.restoreAllMocks(); });
|
afterEach(async () => { vi.useRealTimers(); await fixture.cleanup(); vi.restoreAllMocks(); });
|
||||||
const names = ['t6_zombie_town_1', 't6_zombie_town_2'];
|
const names = ['t6_zombie_town_1', 't6_zombie_town_2'];
|
||||||
const settings = { game: 't6', mapSlug: 'town', maximum: 2, mode: 'create-destroy', emptyMinutes: 1, cooldownMinutes: 1 };
|
const settings = { game: 't6', mapSlug: 'town', maximum: 2, mode: 'create-destroy', emptyMinutes: 1 };
|
||||||
|
|
||||||
async function setup(gameWide = false) {
|
async function setup(gameWide = false) {
|
||||||
const servers = gameWide ? [...names, ...zombieMaps('t6').filter(m => m.slug !== 'town').map(m => `t6_zombie_${m.slug}_1`)] : names;
|
const servers = gameWide ? [...names, ...zombieMaps('t6').filter(m => m.slug !== 'town').map(m => `t6_zombie_${m.slug}_1`)] : names;
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ let fixture: Fixture;
|
|||||||
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
|
||||||
afterEach(async () => { await fixture.cleanup(); });
|
afterEach(async () => { await fixture.cleanup(); });
|
||||||
const town = 't6_zombie_town_1', farm = 't6_zombie_farm_1';
|
const town = 't6_zombie_town_1', farm = 't6_zombie_farm_1';
|
||||||
const policy: ScalingPolicy = { ...scalingSettings({ game: 't6', mapSlug: 'town', maximum: 4, mode: 'create-destroy' }), id: 'policy', revision: 1, nextActionAt: 0, pausedReason: null, updatedAt: '' };
|
const policy: ScalingPolicy = { ...scalingSettings({ game: 't6', mapSlug: 'town', maximum: 4, mode: 'create-destroy' }), id: 'policy', revision: 1, pausedReason: null, updatedAt: '' };
|
||||||
|
|
||||||
it('separates variants, deduplicates pending identities, and initializes additive schema', async () => {
|
it('separates variants, deduplicates pending identities, and initializes additive schema', async () => {
|
||||||
await prepareNativeHost([town, farm]);
|
await prepareNativeHost([town, farm]);
|
||||||
|
|||||||
Reference in New Issue
Block a user