diff --git a/docs/guides/autoscaling.md b/docs/guides/autoscaling.md index 83d80b6..aaa3e27 100644 --- a/docs/guides/autoscaling.md +++ b/docs/guides/autoscaling.md @@ -10,7 +10,7 @@ T5 game-wide policies cover the ten standard Zombies maps; T6 policies cover nin Choose **Add policy**, select the game and scope, and enter an explicit maximum. The initial editor selects Off, with a minimum of one running lobby, one empty spare, and a 15-minute empty interval. Counts must be between 1 and 20; the minimum and spare target cannot exceed the maximum. -Choose **Save policy** to apply changes directly. Saving checks the current fleet, policy revision, ownership conflicts, and host readiness. If saving is blocked, the form keeps your settings and shows the reason. Saved policies show lobby counts, key availability, and the current status. An active job appears once with its latest stage and a **View job** link. Use **Lobbies** for per-map spare coverage and occupancy evidence, **Activity** for decision changes grouped by job, and **Metrics** for observed capacity and provisioning measurements. +Choose **Save policy** to apply changes directly. Saving checks the current fleet, policy revision, ownership conflicts, and host readiness. If saving is blocked, the form keeps your settings and shows the reason. Saved policies show lobby counts, key availability, and the current status. An active job appears once with its latest stage and a **View job** link. Use **Lobbies** for per-map spare coverage and occupancy evidence and **Activity** for decision changes grouped by job. [**Metrics**](metrics.md) shows observed lobby counts and empty spare hours, plus autoscaling additions, removals, failed or cancelled operations, and average/maximum request-to-ready time for the selected period, game, and map. The policy card retains its separate 31-day measurement totals, including spare deficits. | Autoscaling | Behavior | | ----------- | ----------------------------------------------------------------------------------------------------- | @@ -68,7 +68,7 @@ The production observation period and live pilot have not been run as part of im 1. Select one or two representative maps and record the baseline lineup, keys, versions, host CPU and memory, and player-hours. 2. Select On after resolving blockers and checking agent capability. Set the minimum, maximum, and spare target, then choose Save policy. -3. Monitor request-to-ready time from Provisioning, browser visibility, successful external joins, empty lobby-hours, spare deficits, and removals. Increase the spare target if creation latency leaves new groups without a suitable lobby. +3. Use Metrics to monitor request-to-ready time, empty lobby-hours, and autoscaling removals for the selected period and scope. Check the policy card's 31-day spare deficits and individual Provisioning operations for delays; verify browser visibility and successful external joins separately. Increase the spare target if creation latency leaves new groups without a suitable lobby. 4. Expand one map at a time when spare availability and gameplay meet your requirements. Pause if they regress. The page retains up to 31 days of decision evidence and observations, with the 30 most recent decision changes shown. Repeated unchanged evaluations update the decision's last-seen time instead of adding a row. Unresolved intent survives retention. Provisioning history retains the operation and policy backlink separately. diff --git a/docs/guides/metrics.md b/docs/guides/metrics.md index 3e955a5..89735a9 100644 --- a/docs/guides/metrics.md +++ b/docs/guides/metrics.md @@ -26,10 +26,35 @@ history or the original installation date. Background refresh keeps the last successful report visible on failure and offers Retry. Returning to a visible tab refreshes immediately. Game and map filters apply to the whole report, including comparisons, signals, and the heatmap. Map choices -preserve the exact recorded game/map pair and include retained occupancy and match -history. Range and scope are shareable through the URL; refreshes preserve table +preserve the exact recorded game/map pair and include retained occupancy, match, +and autoscaling Provisioning history. Range and scope are shareable through the URL; refreshes preserve table controls and chart selection. +## Autoscaling operations + +The **Autoscaling** section follows the selected period, game, and map and links +to the policies. **Lobbies added** and **Lobbies removed** count successful create +and remove operations, including completions with warnings. Failed and cancelled +operations appear separately. Counts come from retained Provisioning records +linked to autoscaling, independently of occupancy samples; manual operations and +changes in sampled lobby counts do not contribute. + +An operation belongs to the period containing its finish time, including both +endpoints. A request made before the period still counts if it finishes within it; +pending operations and finishes outside the period do not count. **Request to +ready** shows the average and maximum request-to-finish duration of successful +creates with valid timestamps, including queue time. Each create has equal weight. +Without valid completed creates, these timings are unavailable, not zero. Verified +readiness does not establish browser visibility or successful external joins. + +Scope uses the operation's recorded server name, which encodes its game and map +variant. Game-wide policies do not combine different variants when filtering by +map. Friendly map names, slugs, and unambiguous canonical map names are supported; +an ambiguous engine label such as `zm_transit` is not assigned to a variant. +History survives server removal and policy deletion, subject to Provisioning +retention. Missing or purged operations are not estimated. The policy card's +31-day spare-deficit measurements remain separate from this section. + ## Matches and groups **Matches started** counts retained matches whose recorded start falls within the @@ -199,3 +224,10 @@ map durations and coverage. `autoscalingPaused` reports the current global pause separately from each policy's own `pausedReason`. Policy edits and global pause changes invalidate cached metrics reports, as do changes to the match and player participation data used by the report. + +`autoscaling` contains `added`, `removed`, `failed`, `cancelled`, `readinessSamples`, +`readinessMeanSeconds`, and `readinessMaxSeconds`. Counts use `finished_at` in the +selected window; readiness uses `finished_at - started_at` for successful creates +with nonnegative, valid durations. Both timing values are `null` when there are +no readiness samples. Changes to Provisioning outcomes or autoscaling source +links invalidate cached reports, including when no new occupancy is collected. diff --git a/src/lib/metrics.ts b/src/lib/metrics.ts index b859970..7dcc91f 100644 --- a/src/lib/metrics.ts +++ b/src/lib/metrics.ts @@ -131,6 +131,18 @@ export interface PooledAvailability { totalMaps: number; } +export interface AutoscalingMetrics { + /** Retained Provisioning outcomes whose finish time is within the selected window. */ + added: number; + removed: number; + failed: number; + cancelled: number; + /** Successful creates with valid request/finish timestamps; includes queue time. */ + readinessSamples: number; + readinessMeanSeconds: number | null; + readinessMaxSeconds: number | null; +} + export interface MetricsData { days: MetricsDays; since: number; @@ -150,6 +162,7 @@ export interface MetricsData { /** Distinct recorded names sampled in this scope/window, including zero elapsed time. */ sampledServerCount: number; matches: MatchMetrics; + autoscaling: AutoscalingMetrics; maps: MapMetrics[]; points: MetricPoint[]; previous: PreviousMetrics; diff --git a/src/lib/server/db/migrations.ts b/src/lib/server/db/migrations.ts index 46562ee..dc3bf14 100644 --- a/src/lib/server/db/migrations.ts +++ b/src/lib/server/db/migrations.ts @@ -8,6 +8,7 @@ import { installPerformanceIndexes } from './performance-indexes.js'; import { installMetricsMatchRevisions, installMetricsPolicyRevisions, + installMetricsProvisioningRevisions, installReportRevisions } from '../reports/revisions.js'; import { enableIncrementalVacuum, retentionClocks } from './retention-clocks.js'; @@ -79,6 +80,11 @@ export const migrations: readonly Migration[] = [ version: 14, name: 'metrics match report revisions', run: installMetricsMatchRevisions + }, + { + version: 15, + name: 'metrics provisioning report revisions', + run: installMetricsProvisioningRevisions } ]; diff --git a/src/lib/server/fleet/metrics-autoscaling.ts b/src/lib/server/fleet/metrics-autoscaling.ts new file mode 100644 index 0000000..8d29fa9 --- /dev/null +++ b/src/lib/server/fleet/metrics-autoscaling.ts @@ -0,0 +1,89 @@ +import { db } from '../db/index.js'; +import { parseServerName, zombieMaps } from '../../map-aliases.js'; +import type { AutoscalingMetrics, MetricsScope } from '../../metrics.js'; + +/** Provisioning sources survive policy/decision deletion and server removal. */ +export function readAutoscalingMetrics(scope: MetricsScope, since: number, now: number) { + // Group in SQLite to avoid transferring every operation. IN deduplicates source + // links, so one operation always counts once. Keep older groups for scope options. + const groups = db() + .prepare( + `WITH operations AS ( + SELECT server, action, state, finished_at >= ? AS in_window, + ROUND((julianday(finished_at)-julianday(started_at))*86400000)/1000.0 AS seconds + FROM server_deployments + WHERE id IN (SELECT operation_id FROM provisioning_sources WHERE kind='autoscaling') + AND finished_at <= ? AND state IN ('done','warning','failed','cancelled') + ), measured AS ( + SELECT *, CASE WHEN in_window AND action='create' + AND state IN ('done','warning') AND seconds >= 0 THEN seconds END AS readiness + FROM operations + ) + SELECT server, + SUM(in_window AND action='create' AND state IN ('done','warning')) AS added, + SUM(in_window AND action='destroy' AND state IN ('done','warning')) AS removed, + SUM(in_window AND state='failed') AS failed, + SUM(in_window AND state='cancelled') AS cancelled, + COUNT(readiness) AS readinessSamples, + COALESCE(SUM(readiness),0) AS readinessSeconds, + MAX(readiness) AS readinessMaxSeconds + FROM measured GROUP BY server` + ) + .all(new Date(since).toISOString(), new Date(now).toISOString()) as (Omit< + AutoscalingMetrics, + 'readinessMeanSeconds' + > & { server: string; readinessSeconds: number })[]; + const summary: AutoscalingMetrics = { + added: 0, + removed: 0, + failed: 0, + cancelled: 0, + readinessSamples: 0, + readinessMeanSeconds: null, + readinessMaxSeconds: null + }; + const maps = new Map(); + let readinessSeconds = 0; + for (const group of groups) { + // Autoscaling names encode the selected variant. Never use today's server + // registration or the policy scope (which may cover every map in a game). + const parsed = parseServerName(group.server); + const catalog = zombieMaps(parsed?.game ?? null); + const entry = catalog.find( + (map) => + map.slug === parsed?.mapSlug || + map.friendly.toLowerCase().replace(/[^a-z0-9]/g, '') === parsed?.mapSlug + ); + if (parsed && entry) { + const pair = { game: parsed.game, map: entry.friendly }; + maps.set(JSON.stringify(pair), pair); + } + if (scope.game !== null && parsed?.game !== scope.game) continue; + if (scope.map !== null) { + const label = scope.map.toLowerCase(); + // A canonical T6 engine map can describe several variants; do not + // silently combine Town, Farm, Bus Depot and TranZit for that label. + const matches = catalog.filter( + (map) => + map.friendly.toLowerCase() === label || + map.slug === label || + map.canonical === label + ); + if (!entry || matches.length !== 1 || matches[0].slug !== entry.slug) continue; + } + summary.added += group.added; + summary.removed += group.removed; + summary.failed += group.failed; + summary.cancelled += group.cancelled; + summary.readinessSamples += group.readinessSamples; + readinessSeconds += group.readinessSeconds; + if (group.readinessMaxSeconds !== null) + summary.readinessMaxSeconds = Math.max( + summary.readinessMaxSeconds ?? 0, + group.readinessMaxSeconds + ); + } + if (summary.readinessSamples > 0) + summary.readinessMeanSeconds = readinessSeconds / summary.readinessSamples; + return { summary, maps: [...maps.values()] }; +} diff --git a/src/lib/server/fleet/metrics.ts b/src/lib/server/fleet/metrics.ts index ece64f1..1a9ce11 100644 --- a/src/lib/server/fleet/metrics.ts +++ b/src/lib/server/fleet/metrics.ts @@ -2,6 +2,7 @@ import { fleetSettings } from '../settings.js'; import { db } from '../db/index.js'; import { ONLINE_MAX_MS } from '../../server-freshness.js'; import { zombieMaps } from '../../map-aliases.js'; +import { readAutoscalingMetrics } from './metrics-autoscaling.js'; import { parseMetricsQuery, type HeatmapCell, @@ -207,10 +208,11 @@ export function readMetrics(input: MetricsDays | MetricsQuery, now = Date.now()) `SELECT DISTINCT game, map_name AS map FROM leaderboard_matches WHERE started_at <= ?` ) .all(new Date(now).toISOString()) as { game: string; map: string }[]; + const autoscaling = readAutoscalingMetrics(scope, since, now); const compareText = (a: string, b: string) => (a < b ? -1 : a > b ? 1 : 0); const availableMaps = [ ...new Map( - [...groups, ...matchMaps].map(({ game, map }) => [ + [...groups, ...matchMaps, ...autoscaling.maps].map(({ game, map }) => [ JSON.stringify([game, map]), { game, map } ]) @@ -514,6 +516,7 @@ export function readMetrics(input: MetricsDays | MetricsQuery, now = Date.now()) summary, running, matches: matchMetrics(totalMatches), + autoscaling: autoscaling.summary, sampledServerCount: sampledServers.size, maps, points, diff --git a/src/lib/server/reports/revisions.ts b/src/lib/server/reports/revisions.ts index b26aa70..661772a 100644 --- a/src/lib/server/reports/revisions.ts +++ b/src/lib/server/reports/revisions.ts @@ -1,5 +1,26 @@ import type Database from 'better-sqlite3'; +/** Provisioning outcomes remain report inputs even after the policy is removed. */ +export function installMetricsProvisioningRevisions(handle: Database.Database) { + for (const table of ['server_deployments', 'provisioning_sources']) + for (const action of ['INSERT', 'UPDATE', 'DELETE']) { + const versions = + action === 'INSERT' ? ['NEW'] : action === 'DELETE' ? ['OLD'] : ['OLD', 'NEW']; + const fields = ['server', 'action', 'state', 'started_at', 'finished_at']; + const condition = + table === 'provisioning_sources' + ? versions.map((row) => `${row}.kind='autoscaling'`).join(' OR ') + : `EXISTS (SELECT 1 FROM provisioning_sources WHERE operation_id=${action === 'DELETE' ? 'OLD' : 'NEW'}.id AND kind='autoscaling')${ + action === 'UPDATE' + ? ` AND (${fields.map((field) => `OLD.${field} IS NOT NEW.${field}`).join(' OR ')})` + : '' + }`; + handle.exec(`CREATE TRIGGER IF NOT EXISTS report_metrics_${table}_${action} + AFTER ${action} ON ${table} WHEN ${condition} + BEGIN UPDATE report_revisions SET revision=revision+1 WHERE topic='metrics'; END;`); + } +} + /** Match demand includes active and interrupted matches, independent of leaderboard eligibility. */ export function installMetricsMatchRevisions(handle: Database.Database) { for (const table of ['leaderboard_matches', 'leaderboard_match_players']) diff --git a/src/lib/ui/components/metrics/MetricsAutoscaling.svelte b/src/lib/ui/components/metrics/MetricsAutoscaling.svelte new file mode 100644 index 0000000..1e5ac42 --- /dev/null +++ b/src/lib/ui/components/metrics/MetricsAutoscaling.svelte @@ -0,0 +1,70 @@ + + + +
+

Autoscaling

+ Open policies +
+
+ + + + + + +
+

+ {#if outcomes === 0} + No completed, failed or cancelled autoscaling operations recorded in this period and + scope. + {:else} + Autoscaling operations finished in this period and scope. Added and removed counts + include successful operations only. + {/if} + {#if metrics.readinessSamples > 0} + Readiness uses {countLabel(metrics.readinessSamples, 'completed create')} with recorded timing, + including queue time. + {:else} + Request-to-ready time is unavailable without completed creates with recorded timing. + {/if} +

+ + + diff --git a/src/lib/ui/metrics-refresh.ts b/src/lib/ui/metrics-refresh.ts index d314d13..96512f6 100644 --- a/src/lib/ui/metrics-refresh.ts +++ b/src/lib/ui/metrics-refresh.ts @@ -24,6 +24,14 @@ const matches = (value: unknown) => ) && (value.averageGroupSize === null || nonnegative(value.averageGroupSize)) && (value.soloShare === null || (nonnegative(value.soloShare) && value.soloShare <= 100)); +const autoscaling = (value: unknown) => + record(value) && + ['added', 'removed', 'failed', 'cancelled', 'readinessSamples'].every( + (key) => nonnegative(value[key]) && Number.isInteger(value[key]) + ) && + ['readinessMeanSeconds', 'readinessMaxSeconds'].every( + (key) => value[key] === null || nonnegative(value[key]) + ); const policy = (value: unknown) => value === null || (record(value) && @@ -40,7 +48,8 @@ function validReport(value: unknown): value is MetricsData { !numeric(value.since) || !totals(value.summary) || !running(value.running) || - !matches(value.matches) + !matches(value.matches) || + !autoscaling(value.autoscaling) ) return false; if ( diff --git a/src/routes/metrics/+page.svelte b/src/routes/metrics/+page.svelte index bc25e15..025765f 100644 --- a/src/routes/metrics/+page.svelte +++ b/src/routes/metrics/+page.svelte @@ -11,6 +11,7 @@ import SectionHeader from '$ui/components/primitives/SectionHeader.svelte'; import Card from '$ui/components/primitives/Card.svelte'; import MetricsSummary from '$ui/components/metrics/MetricsSummary.svelte'; + import MetricsAutoscaling from '$ui/components/metrics/MetricsAutoscaling.svelte'; import MetricsHeatmap from '$ui/components/metrics/MetricsHeatmap.svelte'; import Tabs from '$ui/components/primitives/Tabs.svelte'; import Button from '$ui/components/primitives/Button.svelte'; @@ -252,6 +253,7 @@ {/if}

{scopeLabel} · {format.period(report.since)} – {format.period(report.now)}

+ {#if captured === 0} +

+ Autoscaling counts retained Provisioning operations whose finish time falls within this + window, including both endpoints. Successful creates and removals include operations + completed with warnings; failed and cancelled operations are counted separately. Pending + operations are excluded. Request-to-ready time measures completed creates from request + through verified readiness, including queue time, and does not establish browser + visibility or successful joins. Game and map scope use the operation’s recorded server + name, preserving map variants and history after server or policy removal. +

Each periodic observation covers at most {ONLINE_MAX_MS / 1000} seconds. Longer collection gaps are excluded. Occupancy history defaults to 31 days; saved match retention is configured diff --git a/tests/unit/database-migrations.test.ts b/tests/unit/database-migrations.test.ts index 4d9bdf2..e8f35c3 100644 --- a/tests/unit/database-migrations.test.ts +++ b/tests/unit/database-migrations.test.ts @@ -205,3 +205,41 @@ it('adds match report invalidation on upgrade without invalidating for score hea expect(revision()).toBeGreaterThan(renamed); expect(upgraded.prepare('SELECT * FROM leaderboard_match_players').all()).toEqual([]); }); + +it('adds provisioning report invalidation at version 14 without invalidating for progress updates', async () => { + const { db, closeDb } = await import('../../src/lib/server/db/index.js'); + const handle = db(); + for (const table of ['server_deployments', 'provisioning_sources']) + for (const action of ['INSERT', 'UPDATE', 'DELETE']) + handle.exec(`DROP TRIGGER report_metrics_${table}_${action}`); + handle.pragma('user_version = 14'); + closeDb(); + const upgraded = db(); + const revision = () => + ( + upgraded + .prepare("SELECT revision FROM report_revisions WHERE topic='metrics'") + .get() as { revision: number } + ).revision; + const before = revision(); + upgraded.exec(`INSERT INTO server_deployments + (id,server,action,state,stage,started_at,triggered_by) + VALUES('operation','t5_zombie_moon_1','create','waiting','Queued', + '2026-10-10T12:00:00.000Z','test')`); + expect(revision()).toBe(before); + upgraded.exec(`INSERT INTO provisioning_sources(operation_id,kind,source_id,server) + VALUES('operation','autoscaling','policy','t5_zombie_moon_1')`); + expect(revision()).toBeGreaterThan(before); + const linked = revision(); + upgraded.exec( + "UPDATE server_deployments SET stage='Still waiting',observed_at='2026-10-10T12:00:15.000Z'" + ); + expect(revision()).toBe(linked); + upgraded.exec( + "UPDATE server_deployments SET state='done',finished_at='2026-10-10T12:01:12.000Z'" + ); + expect(revision()).toBeGreaterThan(linked); + const completed = revision(); + upgraded.exec('DELETE FROM server_deployments'); + expect(revision()).toBeGreaterThan(completed); +}); diff --git a/tests/unit/metrics-autoscaling.test.ts b/tests/unit/metrics-autoscaling.test.ts new file mode 100644 index 0000000..a495931 --- /dev/null +++ b/tests/unit/metrics-autoscaling.test.ts @@ -0,0 +1,189 @@ +import { afterEach, beforeEach, expect, it, vi } from 'vitest'; +import { newFixture, type Fixture } from '../helpers/fixture.js'; +import { parseMetricsQuery } from '../../src/lib/metrics.js'; + +let fixture: Fixture; +const now = Date.UTC(2026, 9, 10, 12); +const day = 86400_000; +beforeEach(async () => { + vi.resetModules(); + fixture = await newFixture([]); +}); +afterEach(async () => { + vi.restoreAllMocks(); + await fixture.cleanup(); +}); + +async function operation( + id: string, + options: { + server?: string; + action?: string; + state?: string; + start?: number | string; + finish?: number | null; + source?: string; + } = {} +) { + const { db } = await import('../../src/lib/server/db/index.js'); + const handle = db(); + const server = options.server ?? 't5_zombie_moon_1'; + const finish = options.finish === undefined ? now - 1000 : options.finish; + const start = options.start ?? (finish ?? now) - 72_000; + handle + .prepare( + `INSERT INTO server_deployments + (id,server,server_deleted_at,action,state,stage,started_at,finished_at,triggered_by) + VALUES(?,?,? ,?,?,'test',?,?,'test')` + ) + .run( + id, + server, + new Date(now).toISOString(), + options.action ?? 'create', + options.state ?? 'done', + typeof start === 'string' ? start : new Date(start).toISOString(), + finish === null ? null : new Date(finish).toISOString() + ); + handle + .prepare( + `INSERT INTO provisioning_sources(operation_id,kind,source_id,server) + VALUES(?,?, 'deleted-policy',?)` + ) + .run(id, options.source ?? 'autoscaling', server); + return handle; +} + +it.each([1, 7, 30] as const)( + 'counts outcomes by finish time in the inclusive %i-day window', + async (days) => { + const { readMetrics } = await import('../../src/lib/server/fleet/metrics.js'); + const since = now - days * day; + await operation('before', { finish: since - 1 }); + await operation('at-start', { finish: since, start: since - 60_000 }); + await operation('at-end', { finish: now, start: now - 120_000, state: 'warning' }); + await operation('after', { finish: now + 1 }); + await operation('removed', { action: 'destroy' }); + await operation('removed-warning', { action: 'destroy', state: 'warning' }); + await operation('failed-create', { state: 'failed' }); + await operation('failed-removal', { action: 'destroy', state: 'failed' }); + await operation('cancelled', { state: 'cancelled' }); + await operation('pending', { state: 'awaiting-health', finish: null }); + await operation('interrupted', { state: 'interrupted' }); + await operation('manual', { source: 'maintenance' }); + await operation('rebuild', { action: 'rebuild' }); + const result = readMetrics(days, now); + expect(result.autoscaling).toEqual({ + added: 2, + removed: 2, + failed: 2, + cancelled: 1, + readinessSamples: 2, + readinessMeanSeconds: 90, + readinessMaxSeconds: 120 + }); + expect(result.summary.observed).toBe(0); + expect(result.maps).toEqual([]); + expect(result.scopeOptions.maps).toEqual([{ game: 't5', map: 'Moon' }]); + } +); + +it('weights readiness by completed creates and counts each operation once after policy/server deletion', async () => { + const { readMetrics } = await import('../../src/lib/server/fleet/metrics.js'); + const handle = await operation('one', { start: now - 61_000 }); + await operation('two', { start: now - 121_000 }); + await operation('three', { server: 't6_zombie_town_1', start: now - 181_000 }); + // Several source rows and another consumer do not multiply the result. + handle + .prepare( + `INSERT INTO provisioning_sources(operation_id,kind,source_id,server) + VALUES('one','autoscaling','second-source','t5_zombie_moon_1'), + ('one','maintenance','maintenance-source','t5_zombie_moon_1')` + ) + .run(); + expect(handle.prepare('SELECT * FROM autoscaling_policies').all()).toEqual([]); + expect(handle.prepare('SELECT * FROM autoscaling_decisions').all()).toEqual([]); + expect(handle.prepare('SELECT * FROM servers').all()).toEqual([]); + expect(readMetrics(7, now).autoscaling).toEqual({ + added: 3, + removed: 0, + failed: 0, + cancelled: 0, + readinessSamples: 3, + readinessMeanSeconds: 120, + readinessMaxSeconds: 180 + }); +}); + +it('filters operation identity by game and map variant independently of the policy scope', async () => { + const { readMetrics } = await import('../../src/lib/server/fleet/metrics.js'); + await operation('moon'); + await operation('town', { server: 't6_zombie_town_1', action: 'destroy' }); + await operation('farm', { server: 't6_zombie_farm_1', state: 'failed' }); + await operation('transit', { server: 't6_zombie_transit_1', state: 'cancelled' }); + await operation('older', { server: 't5_zombie_theater_1', finish: now - 8 * day }); + await operation('future', { server: 't6_zombie_tomb_1', finish: now + 1 }); + const report = (game: string, map = '') => + readMetrics(parseMetricsQuery(new URLSearchParams({ days: '7', game, map })), now); + expect(report('t6').autoscaling).toMatchObject({ + added: 0, + removed: 1, + failed: 1, + cancelled: 1 + }); + expect(report('t5', 'Moon').autoscaling.added).toBe(1); + expect(report('t5', 'zombie_moon').autoscaling.added).toBe(1); + for (const label of ['Town Survival', 'town']) { + expect(report('t6', label).autoscaling).toMatchObject({ + added: 0, + removed: 1, + failed: 0, + cancelled: 0 + }); + } + for (const [game, map] of [ + ['t6', 'zm_transit'], + ['t5', 'Town Survival'], + ['t6', 'Unknown'] + ]) + expect(report(game, map).autoscaling).toMatchObject({ + added: 0, + removed: 0, + failed: 0, + cancelled: 0 + }); + const options = report('t6').scopeOptions; + expect(options.maps).toContainEqual({ game: 't5', map: 'Kino Der Toten' }); + expect(options.maps).not.toContainEqual({ game: 't6', map: 'Origins' }); + expect(report('t5', 'Kino Der Toten').autoscaling.added).toBe(0); + expect(readMetrics(30, now).autoscaling.added).toBe(2); +}); + +it('keeps absent readiness null and excludes invalid durations without losing completed counts', async () => { + const { readMetrics } = await import('../../src/lib/server/fleet/metrics.js'); + expect(readMetrics(7, now).autoscaling).toEqual({ + added: 0, + removed: 0, + failed: 0, + cancelled: 0, + readinessSamples: 0, + readinessMeanSeconds: null, + readinessMaxSeconds: null + }); + await operation('negative', { start: now }); + await operation('invalid-start', { start: 'not-recorded' }); + await operation('missing-finish', { finish: null }); + expect(readMetrics(7, now).autoscaling).toMatchObject({ + added: 2, + readinessSamples: 0, + readinessMeanSeconds: null, + readinessMaxSeconds: null + }); + await operation('zero', { start: now, finish: now }); + expect(readMetrics(7, now).autoscaling).toMatchObject({ + added: 3, + readinessSamples: 1, + readinessMeanSeconds: 0, + readinessMaxSeconds: 0 + }); +}); diff --git a/tests/unit/metrics-refresh.test.ts b/tests/unit/metrics-refresh.test.ts index ee14951..7848b94 100644 --- a/tests/unit/metrics-refresh.test.ts +++ b/tests/unit/metrics-refresh.test.ts @@ -17,6 +17,15 @@ const report = (days: number) => ({ summary: totals, running: { average: null, peak: 0, capturedHours: 0 }, matches: emptyMatches, + autoscaling: { + added: 0, + removed: 0, + failed: 0, + cancelled: 0, + readinessSamples: 0, + readinessMeanSeconds: null, + readinessMaxSeconds: null + }, points: [], maps: [], sampledServerCount: 0, @@ -71,6 +80,26 @@ function setup(fetcher: typeof fetch) { } describe('metrics background refresh', () => { + it.each([ + null, + { ...report(7).autoscaling, added: -1 }, + { ...report(7).autoscaling, cancelled: 1.5 }, + { ...report(7).autoscaling, readinessMeanSeconds: '72' }, + { ...report(7).autoscaling, readinessMaxSeconds: -1 } + ])( + 'keeps the previous report when autoscaling measurements are invalid (%j)', + async (autoscaling) => { + const refresh = setup( + vi.fn().mockResolvedValue(Response.json({ ...report(7), autoscaling })) + ); + await refresh.refresh(); + expect(refresh.onData).not.toHaveBeenCalled(); + expect(refresh.onError).toHaveBeenLastCalledWith( + 'The metrics response was incomplete.' + ); + } + ); + it.each([undefined, null, -1, 1.5, '1'])( 'rejects an invalid sampled-server count (%s) and accepts a count without server rows', async (sampledServerCount) => { diff --git a/tests/unit/report-workers.test.ts b/tests/unit/report-workers.test.ts index 3c238f2..f2c9c9b 100644 --- a/tests/unit/report-workers.test.ts +++ b/tests/unit/report-workers.test.ts @@ -158,3 +158,45 @@ it('refreshes policy signals after policy edits and global pauses without new sa db().prepare("DELETE FROM autoscaling_policies WHERE id='moon-policy'").run(); expect((await metricsReport(query)).maps[0].policy).toBeNull(); }); + +it('refreshes autoscaling outcomes after completion, source changes and retention without new samples', async () => { + const { db } = await import('../../src/lib/server/db/index.js'); + const { metricsReport } = await import('../../src/lib/server/reports/index.js'); + const now = Date.UTC(2026, 9, 10, 12); + vi.spyOn(Date, 'now').mockReturnValue(now); + const query = parseMetricsQuery(new URLSearchParams('days=7')); + const report = () => metricsReport(query); + expect((await report()).autoscaling.added).toBe(0); + db() + .prepare( + `INSERT INTO server_deployments + (id,server,action,state,stage,started_at,triggered_by) + VALUES('operation','t5_zombie_moon_1','create','awaiting-health','Waiting',?,'test')` + ) + .run(new Date(now - 72_000).toISOString()); + db().exec(`INSERT INTO provisioning_sources(operation_id,kind,source_id,server) + VALUES('operation','autoscaling','deleted-policy','t5_zombie_moon_1')`); + expect((await report()).autoscaling.added).toBe(0); + db() + .prepare("UPDATE server_deployments SET state='done',finished_at=? WHERE id='operation'") + .run(new Date(now).toISOString()); + expect((await report()).autoscaling).toMatchObject({ + added: 1, + readinessSamples: 1, + readinessMeanSeconds: 72, + readinessMaxSeconds: 72 + }); + db().exec("UPDATE provisioning_sources SET kind='maintenance'"); + expect((await report()).autoscaling.added).toBe(0); + db().exec("UPDATE provisioning_sources SET kind='autoscaling'"); + expect((await report()).autoscaling.added).toBe(1); + db().exec("UPDATE server_deployments SET state='failed'"); + expect((await report()).autoscaling).toMatchObject({ + added: 0, + failed: 1, + readinessSamples: 0 + }); + db().exec('DELETE FROM server_deployments'); + expect((await report()).autoscaling.failed).toBe(0); + expect((await report()).scopeOptions.maps).toEqual([]); +}); diff --git a/tests/unit/ui-metrics-page.test.ts b/tests/unit/ui-metrics-page.test.ts index 3cbd85c..c90b265 100644 --- a/tests/unit/ui-metrics-page.test.ts +++ b/tests/unit/ui-metrics-page.test.ts @@ -3,6 +3,7 @@ import { render } from 'svelte/server'; import type { MetricsData, MapMetrics, MatchMetrics } from '../../src/lib/metrics.js'; import MetricsPage from '../../src/routes/metrics/+page.svelte'; import MetricsSummary from '../../src/lib/ui/components/metrics/MetricsSummary.svelte'; +import MetricsAutoscaling from '../../src/lib/ui/components/metrics/MetricsAutoscaling.svelte'; import MetricsTables from '../../src/lib/ui/components/metrics/MetricsTables.svelte'; import MetricsExplore from '../../src/lib/ui/components/metrics/MetricsExplore.svelte'; import MetricsChart from '../../src/lib/ui/components/metrics/MetricsChart.svelte'; @@ -64,6 +65,15 @@ function report(): MetricsData { scopeFirstSampleAt: now - 14 * 86400000, scopeLatestSampleAt: now, matches: matches(), + autoscaling: { + added: 73, + removed: 48, + failed: 1, + cancelled: 2, + readinessSamples: 73, + readinessMeanSeconds: 72, + readinessMaxSeconds: 168 + }, summary, running: { average: 12, peak: 24, capturedHours: 1 }, maps: Array.from({ length: 24 }, (_, i) => map(i + 1)), @@ -106,6 +116,50 @@ function scope(search: string, stateSearch?: string) { } beforeEach(() => scope('')); +it('shows Provisioning outcomes and readiness with a policy link, even without occupancy', () => { + const data = report(); + const section = render(MetricsAutoscaling, { props: { metrics: data.autoscaling } }).body; + for (const label of [ + 'Lobbies added', + 'Lobbies removed', + 'Failed operations', + 'Cancelled operations' + ]) + expect(section).toContain(label); + for (const value of [73, 48, 1, 2]) expect(section).toContain(`>${value}<`); + expect(section).toContain('72 s'); + expect(section).toContain('168 s'); + expect(section).toContain('href="/autoscaling"'); + expect(section.replace(/\s+/g, ' ')).toContain('73 completed creates with recorded timing'); + data.summary = { observed: 0, active: 0, idle: 0, unknown: 0, playerHours: 0 }; + data.latestSampleAt = null; + const body = render(MetricsPage, { + props: { data: { ...data, user: null, setupRequired: false } } + }).body; + expect(body).toContain('No retained observations yet'); + expect(body).toContain('Lobbies added'); + expect(body).toContain('72 s'); +}); + +it('distinguishes no autoscaling outcomes from unavailable readiness', () => { + const body = render(MetricsAutoscaling, { + props: { + metrics: { + added: 0, + removed: 0, + failed: 0, + cancelled: 0, + readinessSamples: 0, + readinessMeanSeconds: null, + readinessMaxSeconds: null + } + } + }).body.replace(/\s+/g, ' '); + expect(body).toContain('No completed, failed or cancelled autoscaling operations recorded'); + expect(body).toContain('Request-to-ready time is unavailable'); + expect(body).not.toContain('0 s'); +}); + it.each([ [0, 'No captured time in this scope'], [1, 'Waiting for observed time']