From 33da4482a6c6b6dbb20246bc6663e419c272a0c8 Mon Sep 17 00:00:00 2001 From: Josh Wright Date: Sat, 10 Oct 2026 14:48:52 -0400 Subject: [PATCH] Make metrics reflect autoscaling policies (Refs #137) --- docs/development/README.md | 2 +- docs/guides/autoscaling.md | 2 +- docs/guides/metrics.md | 177 +++++++----- scripts/fixture-data.ts | 2 +- src/lib/metrics-insights.ts | 141 ++++++---- src/lib/metrics.ts | 39 ++- src/lib/server/db/migrations.ts | 7 +- src/lib/server/fleet/metrics.ts | 144 ++++++++-- src/lib/server/reports/revisions.ts | 16 ++ .../ui/components/metrics/MetricsChart.svelte | 105 ++++--- .../components/metrics/MetricsExplore.svelte | 59 ++-- .../components/metrics/MetricsSummary.svelte | 45 ++- .../components/metrics/MetricsTables.svelte | 30 +- src/lib/ui/metrics-refresh.ts | 36 ++- src/lib/ui/metrics.ts | 7 + src/routes/metrics/+page.svelte | 41 ++- tests/unit/database-migrations.test.ts | 29 ++ tests/unit/metrics-insights.test.ts | 182 +++++++++--- tests/unit/metrics-refresh.test.ts | 68 ++++- tests/unit/metrics.test.ts | 265 ++++++++++++++++-- tests/unit/report-workers.test.ts | 56 ++++ tests/unit/seed-fixture.test.ts | 5 +- tests/unit/ui-metrics-page.test.ts | 167 +++++++++-- tests/unit/ui-metrics.test.ts | 9 + 24 files changed, 1264 insertions(+), 370 deletions(-) diff --git a/docs/development/README.md b/docs/development/README.md index 250466e..6587e6f 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -60,7 +60,7 @@ snapshots naturally show stale observations when viewed against today's clock. The fixture contains 3 hosts, 12 canonical Black Ops I/II servers, and 500 match records: 494 historical and 6 live. It includes solo through four-player squads, unknown starting counts, enough players and matches to exercise pagination, -rosters, seven days of utilization and capacity samples, and retained Activity +rosters, seven days of occupancy samples with unknown player limits, and retained Activity from campaigns, betas, host operations, maintenance, recovery, and autoscaling. The fleet has 6 live matches, 2 empty lobbies, a preparing server, and 3 problems: a stopped server, a server not responding, and a server on the offline host. diff --git a/docs/guides/autoscaling.md b/docs/guides/autoscaling.md index a745420..83d80b6 100644 --- a/docs/guides/autoscaling.md +++ b/docs/guides/autoscaling.md @@ -73,7 +73,7 @@ The production observation period and live pilot have not been run as part of im The page retains up to 31 days of decision evidence and observations, with the 30 most recent decision changes shown. Repeated unchanged evaluations update the decision's last-seen time instead of adding a row. Unresolved intent survives retention. Provisioning history retains the operation and policy backlink separately. -Measurement totals exclude collection gaps. Request-to-verified-readiness summaries include queue time and only completed creates; they do not establish browser visibility or successful joins. “Time with players” in Metrics can rise simply because idle lobbies leave its denominator. Use host CPU and memory measurements alongside player-hours and spare availability when assessing savings. +Measurement totals exclude collection gaps. Request-to-verified-readiness summaries include queue time and only completed creates; they do not establish browser visibility or successful joins. [Metrics](metrics.md) shows average and peak lobby counts, empty spare hours, and map time with no reported empty lobby. Its autoscaling signals compare historical counts with current policy targets. Use host CPU and memory measurements alongside player-hours and spare availability when assessing savings. ## API and implementation diff --git a/docs/guides/metrics.md b/docs/guides/metrics.md index df312e3..ecf33ec 100644 --- a/docs/guides/metrics.md +++ b/docs/guides/metrics.md @@ -5,98 +5,137 @@ Choose the past 24 hours, 7 days, or 30 days. The page refreshes every minute. ## Overview and filters -The overview shows **Time with players** (occupied server-hours divided by known -server-hours) and **Player-hours**. Known and unknown status time remain visible -alongside the totals. **Last collected** is the newest recorded batch, including -unknown checks; **Report refreshed** is when this report was calculated. Collection -stops when no servers are registered. Earliest retained observations do not prove -continuous history or the original installation date. +**Lobbies running** shows the average and peak number of recorded lobbies in the +selected scope. The average divides captured lobby-hours, including unknown +status, by captured wall time. Empty spares are deliberate supply for new groups: +the card shows their hours and share of all captured lobby-hours, rather than +treating them as failed utilization. Unknown status can include lobbies starting +or unavailable, so this count does not guarantee every lobby was ready to join. + +**Player-hours** integrates reported player counts over time. **No empty lobby** +shows the share of known map time when no tracked lobby reported zero players. +Each game/map is measured separately, then its durations are pooled. This is not +elapsed fleet time or a count of rejected joins. A partly occupied Zombies lobby +does not count as an empty spare, even if player slots remain. + +**Last collected** is the newest recorded batch, including unknown checks; +**Report refreshed** is when this report was calculated. Collection stops when no +servers are registered. Earliest retained observations do not prove continuous +history or the original installation date. Background refresh keeps the last successful report visible on failure and offers -Retry. Returning to a visible tab refreshes immediately. The selected range is -stored in the URL, and refreshes preserve table controls and chart selection. - -Game and map filters apply to the whole report, including its comparisons, -recommendations, and heatmap. Map choices preserve the exact recorded game/map -pair. Range and scope are shareable through the URL. Table searches remain local. +Retry. Returning to a visible tab refreshes immediately. Game and map filters apply +to the whole report, including comparisons, signals, and the heatmap. Map choices +preserve the exact recorded game/map pair. Range and scope are shareable through +the URL; refreshes preserve table controls and chart selection. ## Timeline and tables -The timeline defaults to **Share of captured time**: each stack includes occupied, -idle, and unknown server time, normalized to 100%. This denominator differs from the -headline utilization percentage, which excludes unknown time. Switch to **Hours** -to compare absolute server time. Periods without captured time remain gaps. -The timeline has labelled axes and marks partial edge periods. Select a period, -use Previous/Next, or select a bar to inspect its exact interval and values. The -time table supplies the same figures for comparison and assistive technology. -Zero captured time is distinct from an observed empty server. Short durations use -seconds or minutes so a real observation is not rounded to zero. +The timeline defaults to **Average lobbies running**, split into **With players**, +**Empty spare**, and **Unknown**. Each bucket divides those lobby-hours by its +captured wall time, so changing lobby counts remain visible. Collection gaps are +excluded from that denominator. Switch to the share view to compare proportions, +or hours to compare absolute lobby time. Periods without captured time remain gaps. -Map and server tables support numeric sorting and local search. Their searches -apply only to the named table. Known-capacity duration stays next to all-full time. -Server identities retain the technical name; rows describe history grouped by -name, which may include different instances if a name was reused. **Open current -server** appears only when that name is currently registered; its destination -represents the current registration rather than every instance in the history. +The timeline has labelled axes and marks partial edge periods. Select a bar or use +Previous/Next to inspect its exact interval and values. The time table supplies +the same figures for comparison and assistive technology. Zero captured time is +distinct from an observed empty lobby. Short durations use seconds or minutes so +a real observation is not rounded to zero. -## Comparisons and hosting experiments +Map and instance tables support numeric sorting and local search. The map table's +**No empty lobby** column uses occupancy alone and retains the known map-time +denominator in its details. Unknown map availability stays distinct from zero +time without a spare. Evidence can be limited by short observations or unknown +occupancy, never by missing player limits. -**Explore this period** adds experimental decision aids below the main timeline: +Instance rows describe history grouped by server name, which can include different +instances if a name was reused. **Open current server** appears only when that name +is currently registered; its destination represents the current registration. -- **Map capacity pressure** pools all-full map-hours divided by known-capacity - map-hours across the selected map groups. Overlapping map intervals are counted - separately. The score is not elapsed fleet time or a fleet health score; its - denominator and contributing map count remain visible. -- **Previous-period comparisons** use the preceding equally long window with - identical filters. Utilization changes are percentage points; player-hours - changes are relative percentages. A previous observed zero can show **New**. - Missing evidence remains unavailable. Partial retained history is labelled; - a 30-day comparison usually has little previous history because default retention is - 31 days. Both periods' known time is shown. Bounds cannot establish that - collection was uninterrupted, and changes are not adjusted for fleet size. -- **Activity heatmap** groups captured time by weekday and hour in its displayed - timezone. Color represents occupied / known server time. Exact cell values - include unknown time and contributing dates. No-data cells differ from observed - zero utilization; repeated DST hours contribute their actual elapsed time. -- **Hosting experiments** are generated automatically from the selected map - observations. They are advisory and never create, restart, or destroy a server. +## Comparisons and activity -Hosting Rules v1 deliberately offer experiments even with small samples: +Previous-period comparisons use the preceding equally long window with identical +filters. Average lobby counts compare recorded supply, while player-hours changes +are relative percentages. A previous observed zero can show **New**. Missing +evidence remains unavailable. Partial retained history is labelled; a 30-day +comparison usually has little previous history because default retention is +31 days. Collection gaps are excluded, and comparisons do not prove uninterrupted +collection or adjust player-hours for fleet size. -| Recorded condition | Suggested experiment | -| ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- | -| At least 20% of known capacity time had every lobby full | Trial one additional lobby during busy hours. | -| Capacity unavailable and players present for at least 80% of known server time | Tentatively trial an extra lobby using occupancy as a proxy. | -| Players present for at most 10% of known time, with under 1% measured pressure or unavailable capacity | Review one idle lobby or a shorter hosting window while preserving access to the map. | -| Occupancy unavailable | Check telemetry/RCON reporting and watch for known player counts. | -| Other observations | Check missing capacity or keep the lineup and watch peaks. | +The activity heatmap groups captured time by weekday and hour in its displayed +timezone. Color represents occupied / known lobby time. Exact cell values include +unknown time and contributing dates. No-data cells differ from observed zero; +repeated DST hours contribute their actual elapsed time. Occupancy share describes +how the fleet was used, but can change simply because a policy adds or removes +spares. Read it alongside player-hours, lobby counts, and empty-lobby availability. -Each suggestion includes evidence and an outcome to watch. Unknown capacity, -less than one hour of relevant known evidence, or substantial unknown captured -status makes the suggestion tentative. These thresholds are product hypotheses; -occupancy does not establish failed joins or a need for additional capacity. -Suggestions use historical observations. Check the current lineup before trying -a change; a retained map may no longer have a hosted lobby. +## Autoscaling signals + +Signals compare each map's observations with its **current** applicable policy. +Game-wide policies apply independently to each supported map variant. Each signal +links to the policy on **Fleet → Autoscaling**; a map without a policy gets a link +to add one. Signals are advisory and never change a policy or a lobby. + +| Recorded condition | Suggested action | +| ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | +| No empty lobby for at least 2% of known map time, mostly at or above the current maximum | Review raising the policy maximum. | +| No empty lobby for at least 2% of known map time, mostly below the current maximum | Review the spare target and check creation time or blockers. | +| An applicable policy is off or paused | Review the policy and its pause or activation conditions. | +| No applicable policy and known occupancy is available | Consider adding an autoscaling policy to keep spares available. | +| Occupancy unavailable | Check telemetry/RCON reporting and wait for known player counts. | +| No shortage signal | Keep watching empty-lobby availability. | + +Evidence includes no-empty map time and how it splits above/below the current +maximum. Less than one hour of known map availability, less than 80% known map +availability, or less than 80% known lobby time makes advice tentative. Player limits are not needed. These +thresholds are advisory rules, not demand forecasts. + +Policy settings may have changed during the selected period. Comparing historical +counts against today's maximum does not establish which settings applied then or +whether autoscaling was enabled. No reported empty lobby also does not prove +failed joins or unmet demand. Review provisioning delays, blockers, host resources, +and game-key availability before changing targets. ## Definitions and collection -- Utilized means a successful status response reported at least one player. -- Idle means a successful response reported zero players. -- Unknown covers failed checks, provisioning, and unavailable containers; it is excluded from utilization percentages. -- Player-hours integrate the reported player count over observed time. Bots are included if the server counts them as players. -- All lobbies full measures map wall time when every tracked lobby in that game/map group has a known capacity and is full. Missing capacity or status makes that interval unknown. This is a capacity-pressure indicator, not a count of rejected joins or unmet demand. +- **With players:** a successful observation reported at least one player. +- **Empty spare:** an observation reported zero players. +- **Unknown:** failed checks, provisioning, or otherwise unavailable occupancy. +- **Player-hours:** reported players × observed time. Bots count if the server includes them. +- **Empty lobby available:** at least one lobby on the game/map reported zero players. +- **No empty lobby:** at least one lobby reported occupancy, and none reported zero. Mixed positive and unknown counts mean no empty lobby was reported; an unknown lobby might actually have been empty. +- **Unknown map availability:** every lobby on the game/map had unknown occupancy. This interval is excluded from the known-map denominator and is never counted as a spare shortage. -Observations are recorded by the existing status worker. Each remains valid until the next collection, up to the 90-second freshness limit. Longer manager outages are excluded. Charts split time at UTC hour/day boundaries; partial periods remain partial. Map variants use the configured variant when it matches the reported map. +Observations come from the existing status worker. Each complete collection stays +valid until the next collection, up to the 90-second freshness limit. Longer gaps +are excluded, not counted as unknown or zero lobbies. Captured wall time includes +intervals with at least one recorded lobby in the selected scope. Unknown lobby +time and unknown map time use different units: several unknown lobbies on one map +contribute several lobby-hours but only one interval of unknown map availability. -Capacity comes from `sv_maxclients` / `sv_maxClients` in the status response. When unavailable, the UI displays unknown rather than assuming a four-player limit. Capacity coverage is the map time for which every tracked lobby reported both occupancy and capacity. +Charts split time at UTC hour/day boundaries; partial periods remain partial. +Map variants use the configured variant when it matches the reported map. Player +capacity may be retained in raw samples, but no metric or signal requires +`sv_maxclients` or an assumed four-player limit. -Collection begins after this feature is installed; previous occupancy cannot be reconstructed. Samples default to 31 days of retention; Admin → Settings can change this period. Cleanup runs during collection and in the background. They survive removal of a server and contain no player identities. The additive database table is created automatically on startup. +Collection begins after this feature is installed; earlier occupancy cannot be +reconstructed. Samples default to 31 days of retention; **Admin → Settings** can +change this period. Cleanup runs during collection and in the background. Samples +survive removal of a server and contain no player identities. ## API -`GET /api/metrics` supplies the same report as the page and disables response +`GET /api/metrics` supplies the same report as the page and disables HTTP response caching. Query parameters are `days` (1, 7, or 30), `game`, `map` (requires game), and `timeZone` (an IANA zone). Invalid ranges and zones fall back to the defaults; invalid scope syntax is normalized. Fleet collection timestamps remain separate from selected-scope evidence timestamps. Collection boundaries are determined before scope filtering, so samples from other groups still end a batch interval. + +`running` contains `average`, `peak`, and `capturedHours`; timeline points include +their own `capturedHours`. Map rows expose `noEmpty`, `availabilityObserved`, and +`availabilityUnknown` durations, plus current `policy` context and +`noEmptyAtMaximum` / `noEmptyBelowMaximum` durations. `pooledAvailability` combines +map durations and coverage. Policy edits and global pause changes invalidate +cached metrics reports. diff --git a/scripts/fixture-data.ts b/scripts/fixture-data.ts index 24c785d..96fbe06 100644 --- a/scripts/fixture-data.ts +++ b/scripts/fixture-data.ts @@ -370,7 +370,7 @@ export function seedFixtureData(handle: Database.Database, now: number): void { } } const metric = handle.prepare( - 'INSERT INTO metric_samples(at,server,game,map,players,capacity) VALUES(?,?,?,?,?,4)' + 'INSERT INTO metric_samples(at,server,game,map,players,capacity) VALUES(?,?,?,?,?,NULL)' ); for (let minute = 0; minute < 7 * 1440; minute++) { for (const [index, server] of fixtureServers.entries()) { diff --git a/src/lib/metrics-insights.ts b/src/lib/metrics-insights.ts index a9be523..2449a76 100644 --- a/src/lib/metrics-insights.ts +++ b/src/lib/metrics-insights.ts @@ -1,113 +1,130 @@ import type { MapMetrics } from './metrics.js'; import { metricDuration, metricPercent } from './ui/metrics.js'; -/** Product hypotheses for advisory experiments; these are not demand forecasts. */ -export const HOSTING_RULES = { - pressureExpansion: 0.2, - occupancyExpansion: 0.8, - idleUtilization: 0.1, - quietPressure: 0.01, - minimumEvidenceHours: 1 +/** Advisory thresholds for reviewing a policy, not demand forecasts or scaling commands. */ +export const POLICY_SIGNAL_RULES = { + noEmptyShare: 0.02, + minimumEvidenceHours: 1, + minimumKnownShare: 0.8 } as const; export interface MetricRecommendation { game: string; map: string; - kind: 'expand' | 'consolidate' | 'capacity' | 'watch'; + kind: 'maximum' | 'spares' | 'policy' | 'reporting' | 'watch'; title: string; reason: string; evidence: string; outcome: string; tentative: boolean; + policySummary: string | null; + href: string | null; + action: string | null; } -/** Apply the same visible rules to all selected maps, including sparse observations. */ +/** Compare captured availability with the current policy; historical settings are not recorded. */ export function metricRecommendations(maps: MapMetrics[]): MetricRecommendation[] { return maps - .map((map) => { - const utilization = map.observed > 0 ? map.active / map.observed : null; - const pressure = map.capacityObserved > 0 ? map.full / map.capacityObserved : null; - const knownShare = - map.observed + map.unknown > 0 ? map.observed / (map.observed + map.unknown) : 0; + .map((map): MetricRecommendation => { + const availability = map.availabilityObserved + map.availabilityUnknown; + const knownShare = availability > 0 ? map.availabilityObserved / availability : 0; + const lobbyHours = map.observed + map.unknown; + const knownLobbyShare = lobbyHours > 0 ? map.observed / lobbyHours : 0; + const noEmptyShare = + map.availabilityObserved > 0 ? map.noEmpty / map.availabilityObserved : null; + const policy = map.policy; const base = { game: map.game, map: map.map, evidence: - `${metricDuration(map.active)} with players / ${metricDuration(map.observed)} known server time (${metricPercent(map.active, map.observed)}). ` + - (pressure === null - ? 'Capacity unavailable.' - : `All lobbies full ${metricDuration(map.full)} / ${metricDuration(map.capacityObserved)} known map time (${metricPercent(map.full, map.capacityObserved)}).`), + `${metricDuration(map.active)} with players / ${metricDuration(map.observed)} known lobby time (${metricPercent(map.active, map.observed)}). ` + + (map.unknown > 0 + ? `${metricDuration(map.unknown)} of lobby time has unknown occupancy. ` + : '') + + (noEmptyShare === null + ? 'Lobby availability is unknown.' + : `No empty lobby reported for ${metricDuration(map.noEmpty)} / ${metricDuration(map.availabilityObserved)} known map time (${metricPercent(map.noEmpty, map.availabilityObserved)}).`) + + (policy && noEmptyShare !== null + ? ` At or above the current maximum (${policy.maximum}): ${metricDuration(map.noEmptyAtMaximum)}; below it: ${metricDuration(map.noEmptyBelowMaximum)}.` + : ''), tentative: - map.observed < HOSTING_RULES.minimumEvidenceHours || - pressure === null || - map.capacityObserved < HOSTING_RULES.minimumEvidenceHours || - knownShare < 0.8 + map.availabilityObserved < POLICY_SIGNAL_RULES.minimumEvidenceHours || + knownShare < POLICY_SIGNAL_RULES.minimumKnownShare || + knownLobbyShare < POLICY_SIGNAL_RULES.minimumKnownShare, + policySummary: policy + ? `Current policy: ${policy.minimum}–${policy.maximum} lobbies, ${policy.spares} spare${policy.spares === 1 ? '' : 's'}` + : null, + href: policy ? `/autoscaling#${encodeURIComponent(policy.id)}` : null, + action: policy ? 'Review policy' : null }; - if (utilization === null) + if (noEmptyShare === null) return { ...base, - kind: 'capacity' as const, + kind: 'reporting', title: - map.unknown > 0 + map.availabilityUnknown > 0 || map.unknown > 0 ? 'Check occupancy reporting' : 'Let observations accumulate', reason: - map.unknown > 0 - ? 'Captured status is unknown. Check a server’s telemetry or RCON diagnostics before choosing a hosting change.' - : 'This map has no elapsed observations in the selected window. Wait for another status check or try a wider range.', - outcome: 'Watch for known server time in the next report.', + map.availabilityUnknown > 0 || map.unknown > 0 + ? 'Captured occupancy is unknown. Check a lobby’s telemetry or RCON diagnostics before choosing a policy change.' + : 'This map has no elapsed availability observations in the selected window. Wait for another status check or try a wider range.', + outcome: 'Watch for known lobby availability in the next report.', tentative: true }; - if (pressure !== null && pressure >= HOSTING_RULES.pressureExpansion) + if (!policy) return { ...base, - kind: 'expand' as const, - title: 'Trial one additional lobby', - reason: 'Recorded lobbies were all full for at least 20% of known capacity time. Try one extra lobby during busy hours.', + kind: 'policy', + title: 'Consider adding an autoscaling policy', + reason: 'This map has no current autoscaling policy. Set a minimum, maximum, and spare target to maintain an empty lobby as players arrive.', outcome: - 'Compare all-full time and use of the additional lobby over the next comparable period.' + 'Choose initial limits from observed lobby use, then compare empty-lobby availability over a comparable period.', + href: '/autoscaling', + action: 'Add policy' }; - if (pressure === null && utilization >= HOSTING_RULES.occupancyExpansion) + if (policy.pausedReason || policy.mode === 'off') return { ...base, - kind: 'expand' as const, - title: 'Trial an extra lobby during busy hours', - reason: 'Players were present for at least 80% of known server time. Capacity is unknown, so this experiment uses occupancy as a proxy, not evidence of saturation.', + kind: 'policy', + title: policy.pausedReason + ? 'Review and resume the autoscaling policy' + : 'Review and enable the autoscaling policy', + reason: policy.pausedReason + ? `The current policy is paused: ${policy.pausedReason}` + : 'The current policy is off, so it cannot maintain the spare target.', outcome: - 'Check whether players use the extra lobby, and measure capacity if the server reports it.', - tentative: true + 'Review the policy and any blockers before enabling it, then observe availability before adjusting its limits.' }; - if ( - utilization <= HOSTING_RULES.idleUtilization && - (pressure === null || pressure < HOSTING_RULES.quietPressure) - ) + if (noEmptyShare >= POLICY_SIGNAL_RULES.noEmptyShare) { + if (map.noEmptyAtMaximum >= map.noEmptyBelowMaximum) + return { + ...base, + kind: 'maximum', + title: 'Consider raising the policy maximum', + reason: `No empty lobby was reported for at least 2% of known map time. At least half of that time had a tracked lobby count at or above the current maximum of ${policy.maximum}; review whether that limit leaves room for the spare target.`, + outcome: + 'Review host resources and create time, then compare empty-lobby availability after a maximum change.' + }; return { ...base, - kind: 'consolidate' as const, - title: 'Trial less idle hosting', - reason: 'Players were present for at most 10% of known time, with less than 1% measured pressure or unavailable capacity. Review one idle lobby or a shorter hosting window; preserve access to this map.', + kind: 'spares', + title: 'Review the spare target and create time', + reason: `No empty lobby was reported for at least 2% of known map time. Most of that time was below the current maximum of ${policy.maximum}; consider raising the spare target from ${policy.spares} and check how long new lobbies take to become ready.`, outcome: - 'Watch for increased all-full time or loss of player-hours, and restore the lineup if the trial hurts availability.' - }; - if (pressure === null) - return { - ...base, - kind: 'capacity' as const, - title: 'Check capacity during busy hours', - reason: 'Occupancy is recorded, but player limits are unavailable. Check capacity reporting while keeping the current lineup.', - outcome: - 'Look for known-capacity time and all-full periods before the next hosting trial.' + 'Review creation delays and blockers, then compare empty-lobby availability after a spare-target change.' }; + } return { ...base, - kind: 'watch' as const, - title: 'Keep the lineup and watch peaks', - reason: 'These observations do not trigger the expansion or idle-hosting rules. Keep the current lineup for another comparison period.', - outcome: 'Watch whether occupied time and measured capacity pressure change.' + kind: 'watch', + title: 'Keep the policy and watch availability', + reason: 'No-empty-lobby time is below the policy review threshold. These observations do not suggest increasing the maximum or spare target.', + outcome: 'Watch for recurring periods without a reported empty lobby.' }; }) .sort((a, b) => { - const priority = { expand: 0, consolidate: 1, capacity: 2, watch: 3 }; + const priority = { maximum: 0, spares: 1, policy: 2, reporting: 3, watch: 4 }; return ( priority[a.kind] - priority[b.kind] || a.map.localeCompare(b.map) || diff --git a/src/lib/metrics.ts b/src/lib/metrics.ts index e66a748..733427d 100644 --- a/src/lib/metrics.ts +++ b/src/lib/metrics.ts @@ -1,4 +1,5 @@ import { DEFAULT_TIME_ZONE, validTimeZone } from './time-zone.js'; +import type { AutoscalingMode } from './autoscaling.js'; export type MetricsDays = 1 | 7 | 30; @@ -58,9 +59,28 @@ export interface ServerMetrics extends Utilization { export interface MapMetrics extends Utilization { game: string; map: string; - /** Map wall time, not summed server time. */ - full: number; - capacityObserved: number; + /** Map wall time, not summed lobby time. Capacity does not affect availability. */ + noEmpty: number; + availabilityObserved: number; + availabilityUnknown: number; + /** Historical lobby counts compared with the current policy's maximum. */ + noEmptyAtMaximum: number; + noEmptyBelowMaximum: number; + policy: { + id: string; + mode: AutoscalingMode; + minimum: number; + maximum: number; + spares: number; + pausedReason: string | null; + } | null; +} + +export interface RunningMetrics { + /** Concurrent tracked lobbies, including unknown player status, during captured scope time. */ + average: number | null; + peak: number; + capturedHours: number; } export interface MetricPoint extends Utilization { @@ -69,12 +89,15 @@ export interface MetricPoint extends Utilization { startAt: number; endAt: number; partial: boolean; + /** Captured scope wall time, excluding collection gaps. */ + capturedHours: number; } export interface PreviousMetrics { since: number; until: number; summary: Utilization; + running: RunningMetrics; firstSampleAt: number | null; lastSampleAt: number | null; /** Retained scope evidence starts late or is absent, not a completeness guarantee. */ @@ -89,11 +112,12 @@ export interface HeatmapCell extends Utilization { distinctDates: number; } -export interface PooledPressure { - /** Share of known map-hours with every tracked lobby full, from 0 to 100. */ +export interface PooledAvailability { + /** Share of known map-hours with no tracked lobby empty, from 0 to 100. */ value: number | null; - fullMapHours: number; + noEmptyMapHours: number; knownMapHours: number; + unknownMapHours: number; coveredMaps: number; totalMaps: number; } @@ -111,10 +135,11 @@ export interface MetricsData { scopeFirstSampleAt: number | null; scopeLatestSampleAt: number | null; summary: Utilization; + running: RunningMetrics; servers: ServerMetrics[]; maps: MapMetrics[]; points: MetricPoint[]; previous: PreviousMetrics; heatmap: { timeZone: string; cells: HeatmapCell[] }; - pooledPressure: PooledPressure; + pooledAvailability: PooledAvailability; } diff --git a/src/lib/server/db/migrations.ts b/src/lib/server/db/migrations.ts index 389a07c..bd4bfaa 100644 --- a/src/lib/server/db/migrations.ts +++ b/src/lib/server/db/migrations.ts @@ -5,7 +5,7 @@ import { ensureAutoscalingSchema } from '../autoscaling/schema.js'; import { ensureRebalancingSchema } from '../rebalancing/schema.js'; import { migrateFileBlobs } from './file-blobs.js'; import { installPerformanceIndexes } from './performance-indexes.js'; -import { installReportRevisions } from '../reports/revisions.js'; +import { installMetricsPolicyRevisions, installReportRevisions } from '../reports/revisions.js'; import { enableIncrementalVacuum, retentionClocks } from './retention-clocks.js'; import { migrateHostSchema } from '../hosts/schema.js'; import { migrateAuthSchema } from '../auth/schema.js'; @@ -65,6 +65,11 @@ export const migrations: readonly Migration[] = [ name: 'incremental vacuum page map', mode: 'autocommit', run: enableIncrementalVacuum + }, + { + version: 13, + name: 'metrics policy report revisions', + run: installMetricsPolicyRevisions } ]; diff --git a/src/lib/server/fleet/metrics.ts b/src/lib/server/fleet/metrics.ts index 77efa0d..5fde9b8 100644 --- a/src/lib/server/fleet/metrics.ts +++ b/src/lib/server/fleet/metrics.ts @@ -1,6 +1,7 @@ import { fleetSettings } from '../settings.js'; import { db } from '../db/index.js'; import { ONLINE_MAX_MS } from '../../server-freshness.js'; +import { zombieMaps } from '../../map-aliases.js'; import { parseMetricsQuery, type HeatmapCell, @@ -10,6 +11,7 @@ import { type MetricsDays, type MetricsQuery, type PreviousMetrics, + type RunningMetrics, type ServerMetrics, type Utilization } from '../../metrics.js'; @@ -41,6 +43,18 @@ function emptyUtilization(): Utilization { return { observed: 0, active: 0, idle: 0, unknown: 0, playerHours: 0 }; } +function emptyRunning(): RunningMetrics { + return { average: null, peak: 0, capturedHours: 0 }; +} + +function addRunning(total: RunningMetrics, count: number, hours: number): void { + if (hours <= 0) return; + const lobbyHours = (total.average ?? 0) * total.capturedHours + count * hours; + total.capturedHours += hours; + total.average = lobbyHours / total.capturedHours; + total.peak = Math.max(total.peak, count); +} + function addObservation(total: Utilization, players: number | null, hours: number): void { if (players === null) total.unknown += hours; else { @@ -71,6 +85,72 @@ export function readMetrics(input: MetricsDays | MetricsQuery, now = Date.now()) const since = now - days * 86400_000; const bucketMs = days === 1 ? 3600_000 : 86400_000; const handle = db(); + // Current settings are context for historical counts, never a reconstruction + // of earlier policy revisions. Avoid loading autoscaling's mutation services. + const globalPaused = + ( + handle.prepare("SELECT value FROM settings WHERE key='autoscaling_paused'").get() as + { value: string } | undefined + )?.value === '1'; + const policies = ( + handle + .prepare( + `SELECT id,game,map_slug,mode,settings,paused_reason,owns_map + FROM autoscaling_policies ORDER BY owns_map DESC,map_slug,id` + ) + .all() as { + id: string; + game: string; + map_slug: string; + mode: NonNullable['mode']; + settings: string; + paused_reason: string | null; + owns_map: number; + }[] + ).map((row) => { + const { minimum, maximum, spares } = JSON.parse(row.settings) as { + minimum: number; + maximum: number; + spares: number; + }; + return { + game: row.game, + mapSlug: row.map_slug, + ownsMap: row.owns_map === 1, + context: { + id: row.id, + mode: row.mode, + minimum, + maximum, + spares, + pausedReason: globalPaused + ? ['Autoscaling is globally paused.', row.paused_reason] + .filter(Boolean) + .join(' ') + : row.paused_reason + } + }; + }); + const policyFor = (game: string, label: string): MapMetrics['policy'] => { + const normalized = label.toLowerCase(); + const catalog = zombieMaps(game); + const named = catalog.find( + (entry) => entry.friendly.toLowerCase() === normalized || entry.slug === normalized + ); + // T6 variants share canonical names; do not choose a variant arbitrarily. + const canonical = catalog.filter((entry) => entry.canonical === normalized); + const slug = named?.slug ?? (canonical.length === 1 ? canonical[0].slug : null); + if (!slug) return null; + const applicable = policies.filter( + (policy) => policy.game === game && (policy.mapSlug === slug || policy.mapSlug === '*') + ); + return ( + applicable.find((policy) => policy.ownsMap)?.context ?? + applicable.find((policy) => policy.mapSlug === slug)?.context ?? + applicable[0]?.context ?? + null + ); + }; // These small grouped results keep options and retained evidence independent // of the chosen scope/window without transferring a second raw history scan. const groups = handle @@ -112,10 +192,12 @@ export function readMetrics(input: MetricsDays | MetricsQuery, now = Date.now()) maps: groups.map(({ game, map }) => ({ game, map })) }; const summary = emptyUtilization(); + const running = emptyRunning(); const previous: PreviousMetrics = { since: since - days * 86400_000, until: since, summary: emptyUtilization(), + running: emptyRunning(), firstSampleAt: null, lastSampleAt: null, partialHistory: false @@ -131,6 +213,7 @@ export function readMetrics(input: MetricsDays | MetricsQuery, now = Date.now()) startAt: start, endAt: end, partial: start !== at || end !== at + bucketMs, + capturedHours: 0, ...emptyUtilization() }); } @@ -211,6 +294,7 @@ export function readMetrics(input: MetricsDays | MetricsQuery, now = Date.now()) if (priorHours > 0) { previous.firstSampleAt ??= at; previous.lastSampleAt = at; + addRunning(previous.running, samples.length, priorHours); for (const sample of samples) addObservation(previous.summary, sample.players, priorHours); } @@ -219,7 +303,11 @@ export function readMetrics(input: MetricsDays | MetricsQuery, now = Date.now()) if (at < since - ONLINE_MAX_MS) return; const start = Math.max(at, since); const hours = Math.max(0, end - start) / 3600_000; - const pressure = new Map(); + addRunning(running, samples.length, hours); + const availability = new Map< + MapMetrics, + { known: boolean; empty: boolean; lobbies: number } + >(); const counts: BatchCounts = { known: 0, active: 0, idle: 0, unknown: 0, players: 0 }; // Split once for the entire batch; observations crossing UTC hour/day // boundaries contribute to both buckets without shifting their duration. @@ -228,7 +316,9 @@ export function readMetrics(input: MetricsDays | MetricsQuery, now = Date.now()) const index = Math.floor((cursor - points[0].at) / bucketMs); const point = points[index]; const stop = Math.min(end, point.endAt); - slices.push({ point, hours: (stop - cursor) / 3600_000 }); + const sliceHours = (stop - cursor) / 3600_000; + point.capturedHours += sliceHours; + slices.push({ point, hours: sliceHours }); cursor = stop; } for (const sample of samples) { @@ -251,8 +341,12 @@ export function readMetrics(input: MetricsDays | MetricsQuery, now = Date.now()) map = { game: sample.game, map: sample.map, - full: 0, - capacityObserved: 0, + noEmpty: 0, + availabilityObserved: 0, + availabilityUnknown: 0, + noEmptyAtMaximum: 0, + noEmptyBelowMaximum: 0, + policy: policyFor(sample.game, sample.map), ...emptyUtilization() }; game.set(sample.map, map); @@ -268,20 +362,24 @@ export function readMetrics(input: MetricsDays | MetricsQuery, now = Date.now()) if (sample.players > 0) counts.active++; else if (sample.players === 0) counts.idle++; } - const known = - sample.players !== null && sample.capacity !== null && sample.capacity > 0; - const full = known && sample.players! >= sample.capacity!; - const previous = pressure.get(map); - pressure.set(map, { - known: known && (previous?.known ?? true), - full: full && (previous?.full ?? true) + const prior = availability.get(map); + availability.set(map, { + known: sample.players !== null || (prior?.known ?? false), + empty: sample.players === 0 || (prior?.empty ?? false), + lobbies: (prior?.lobbies ?? 0) + 1 }); } - for (const [map, state] of pressure) { + for (const [map, state] of availability) { if (state.known) { - map.capacityObserved += hours; - if (state.full) map.full += hours; - } + map.availabilityObserved += hours; + if (!state.empty) { + map.noEmpty += hours; + if (map.policy) { + if (state.lobbies >= map.policy.maximum) map.noEmptyAtMaximum += hours; + else map.noEmptyBelowMaximum += hours; + } + } + } else map.availabilityUnknown += hours; } addHeatmap(counts, start, end); } @@ -328,13 +426,14 @@ export function readMetrics(input: MetricsDays | MetricsQuery, now = Date.now()) previous.firstSampleAt === null; for (let index = 0; index < cells.length; index++) cells[index].distinctDates = cellDates[index].size; - const fullMapHours = maps.reduce((total, map) => total + map.full, 0); - const knownMapHours = maps.reduce((total, map) => total + map.capacityObserved, 0); - const pooledPressure = { - value: knownMapHours > 0 ? (100 * fullMapHours) / knownMapHours : null, - fullMapHours, + const noEmptyMapHours = maps.reduce((total, map) => total + map.noEmpty, 0); + const knownMapHours = maps.reduce((total, map) => total + map.availabilityObserved, 0); + const pooledAvailability = { + value: knownMapHours > 0 ? (100 * noEmptyMapHours) / knownMapHours : null, + noEmptyMapHours, knownMapHours, - coveredMaps: maps.filter((map) => map.capacityObserved > 0).length, + unknownMapHours: maps.reduce((total, map) => total + map.availabilityUnknown, 0), + coveredMaps: maps.filter((map) => map.availabilityObserved > 0).length, totalMaps: maps.length }; return { @@ -345,11 +444,12 @@ export function readMetrics(input: MetricsDays | MetricsQuery, now = Date.now()) scope, scopeOptions, summary, + running, servers, maps, points, previous, heatmap: { timeZone: scope.timeZone, cells }, - pooledPressure + pooledAvailability }; } diff --git a/src/lib/server/reports/revisions.ts b/src/lib/server/reports/revisions.ts index 9e245d3..5839c69 100644 --- a/src/lib/server/reports/revisions.ts +++ b/src/lib/server/reports/revisions.ts @@ -1,5 +1,21 @@ import type Database from 'better-sqlite3'; +/** Current policy settings are inputs to historical metrics advice. */ +export function installMetricsPolicyRevisions(handle: Database.Database) { + for (const table of ['autoscaling_policies', 'settings']) + for (const action of ['INSERT', 'UPDATE', 'DELETE']) { + const versions = + action === 'INSERT' ? ['NEW'] : action === 'DELETE' ? ['OLD'] : ['OLD', 'NEW']; + const condition = + table === 'settings' + ? `WHEN ${versions.map((row) => `${row}.key='autoscaling_paused'`).join(' OR ')}` + : ''; + handle.exec(`CREATE TRIGGER IF NOT EXISTS report_metrics_${table}_${action} + AFTER ${action} ON ${table} ${condition} + BEGIN UPDATE report_revisions SET revision=revision+1 WHERE topic='metrics'; END;`); + } +} + export function installReportRevisions(handle: Database.Database) { handle.exec(`CREATE TABLE IF NOT EXISTS report_revisions (topic TEXT PRIMARY KEY, revision INTEGER NOT NULL DEFAULT 0); INSERT OR IGNORE INTO report_revisions(topic) VALUES ('metrics'), ('leaderboard');`); diff --git a/src/lib/ui/components/metrics/MetricsChart.svelte b/src/lib/ui/components/metrics/MetricsChart.svelte index 430c18b..ad2b9f4 100644 --- a/src/lib/ui/components/metrics/MetricsChart.svelte +++ b/src/lib/ui/components/metrics/MetricsChart.svelte @@ -5,6 +5,7 @@ import { createTimeFormatter } from '$lib/time-zone.js'; import { metricDuration, + metricAverageLobbies, metricNumber, metricPercent, metricPeriodLabels @@ -21,9 +22,10 @@ const contextSearch = $derived( (page.state as { metricsTableSearch?: string }).metricsTableSearch ?? page.url.search ); - const mode = $derived( - new URLSearchParams(contextSearch).get('chartMode') === 'hours' ? 'hours' : 'percent' - ); + const mode = $derived.by(() => { + const value = new URLSearchParams(contextSearch).get('chartMode'); + return value === 'hours' || value === 'percent' ? value : 'average'; + }); const selectedIndex = $derived.by(() => { const existing = points.findIndex((point) => point.at === selectedAt); if (existing >= 0) return existing; @@ -34,7 +36,15 @@ const selectedValue = $derived(selected ? String(selected.at) : ''); const ceiling = $derived.by(() => { if (mode === 'percent') return 100; - const max = Math.max(0, ...points.map((point) => point.observed + point.unknown)); + const max = Math.max( + 0, + ...points.map((point) => + mode === 'average' + ? (metricAverageLobbies(point.observed + point.unknown, point.capturedHours) ?? + 0) + : point.observed + point.unknown + ) + ); if (!max) return 1; const magnitude = 10 ** Math.floor(Math.log10(max)); return Math.ceil(max / (magnitude / 2)) * (magnitude / 2); @@ -55,10 +65,10 @@ const periodLabels = $derived(metricPeriodLabels(points, days, timeZone)); const captured = $derived(selected ? selected.observed + selected.unknown : 0); - function setMode(value: 'hours' | 'percent') { + function setMode(value: 'average' | 'hours' | 'percent') { const url = new URL(page.url); url.search = contextSearch; - if (value === 'percent') url.searchParams.delete('chartMode'); + if (value === 'average') url.searchParams.delete('chartMode'); else url.searchParams.set('chartMode', value); replaceState(url, { ...page.state, metricsTableSearch: url.search }); } @@ -66,12 +76,27 @@ selectedAt = at; const point = points.find((item) => item.at === at); if (!point) return; - announcement = `${format.period(point.startAt)} to ${format.period(point.endAt)}${point.partial ? ', partial period' : ''}. ${point.observed + point.unknown ? `${metricDuration(point.active)} with players, ${metricDuration(point.idle)} idle, ${metricDuration(point.unknown)} unknown.` : 'No captured time.'}`; + announcement = `${format.period(point.startAt)} to ${format.period(point.endAt)}${point.partial ? ', partial period' : ''}. ${pointSummary(point)}`; } function segmentHeight(point: MetricPoint, value: number) { - const denominator = mode === 'percent' ? point.observed + point.unknown : ceiling; + const denominator = + mode === 'percent' + ? point.observed + point.unknown + : mode === 'average' + ? ceiling * point.capturedHours + : ceiling; return denominator > 0 ? (value / denominator) * 100 : 0; } + function averageLabel(point: MetricPoint, value: number) { + const average = metricAverageLobbies(value, point.capturedHours); + return average === null ? '—' : metricNumber(average); + } + function pointSummary(point: MetricPoint) { + if (point.observed + point.unknown === 0) return 'No captured time'; + return mode === 'average' + ? `${averageLabel(point, point.active)} with players · ${averageLabel(point, point.idle)} empty spares · ${averageLabel(point, point.unknown)} unknown on average` + : `${metricDuration(point.active)} with players · ${metricDuration(point.idle)} empty spares · ${metricDuration(point.unknown)} unknown`; + } function move(direction: number) { const next = points[Math.max(0, Math.min(points.length - 1, selectedIndex + direction))]; if (next) selectPeriod(next.at); @@ -98,15 +123,22 @@
-

Server activity over time

+

Lobbies running over time

- {mode === 'percent' - ? 'Share of captured server time, including unknown status.' - : `Server-hours per ${days === 1 ? 'hour' : '24-hour period'}.`} Partial periods contain - less elapsed time. + {mode === 'average' + ? 'Average concurrent lobbies during captured time, including unknown status.' + : mode === 'percent' + ? 'Share of captured lobby time, including unknown status.' + : `Lobby-hours per ${days === 1 ? 'hour' : '24-hour period'}.`} + Partial periods contain less elapsed time.

+
- With playersIdleUnknown* Partial period + With playersEmpty spareUnknown* Partial period
- {mode === 'percent' ? 'Share of captured server time' : 'Server-hours'} + {mode === 'average' + ? 'Average lobbies running' + : mode === 'percent' + ? 'Share of captured lobby time' + : 'Lobby-hours'}
{suggestion.title} @@ -58,17 +57,24 @@

{suggestion.reason}

{suggestion.outcome}

+ {#if suggestion.href && suggestion.action} + {suggestion.action} + {/if}
{/snippet}
-

Hosting insights

-

Historical signals for hosting trials. Check the current lineup before making a change.

+

Autoscaling signals

+

+ Compare recorded empty-lobby availability with each map’s current policy. Current settings + are applied to historical lobby counts; these are not records of the policy active then. No + reported empty lobby does not prove that players were turned away. +

{#if highlighted.length}{#each highlighted as suggestion}{@render finding(suggestion)}{/each} - {:else}

No hosting trials meet the evidence threshold in this period.

{/if} + {:else}

No policy changes meet the evidence threshold in this period.

{/if} {#if limited}

- {limited} of {recommendations.length} maps have limited evidence. Review occupancy reporting, - player limits, and observation duration before changing hosting. + {limited} of {recommendations.length} maps have limited evidence. Review occupancy reporting + and observation duration before changing a policy.

{/if} {#if remaining.length}
Other findings ({remaining.length}) @@ -104,31 +110,33 @@

{/each}
{/if}
- How Rules v1 chooses a suggestion + How policy signals are chosen
  • - Trial an additional lobby when all tracked lobbies are full for at least {HOSTING_RULES.pressureExpansion * - 100}% of known-capacity time. + Review a policy when no empty lobby was reported for at least {POLICY_SIGNAL_RULES.noEmptyShare * + 100}% of known map time. When most of that time had a lobby count at or above + the current maximum, consider raising the maximum. Otherwise, review the spare + target and create time. Equal durations favor reviewing the maximum.
  • - When capacity is unavailable, at least {HOSTING_RULES.occupancyExpansion * 100}% - occupied known time can suggest a tentative additional-lobby trial. Occupancy does - not establish saturation. + An empty lobby has a reported player count of zero. When some lobbies report players + and the rest are unknown, none reports empty; unknown lobbies may still have been + empty. Player limits are not needed for these signals.
  • - Review idle hosting when occupied known time is at most {HOSTING_RULES.idleUtilization * - 100}%, and either measured pressure is below {HOSTING_RULES.quietPressure * - 100}% or capacity is unavailable. + Maps without a current policy suggest adding one. Paused or disabled policies prompt + a review before resuming or enabling them.
  • - Missing occupancy prompts a reporting check. Other cases suggest watching hosting or - checking capacity reporting. + All-unknown occupancy prompts a reporting check. Other cases suggest watching + empty-lobby availability.

- Less than {HOSTING_RULES.minimumEvidenceHours} hour of relevant known time, missing capacity, - or substantial unknown status marks advice as tentative. These are trial policies, not validated - demand forecasts. Collection gaps are omitted from all rules. + Less than {POLICY_SIGNAL_RULES.minimumEvidenceHours} hour of known map availability, or less + than {POLICY_SIGNAL_RULES.minimumKnownShare * 100}% known map availability or lobby + occupancy, marks a signal as tentative. These signals are advisory, not demand + forecasts. Collection gaps are omitted from all rules.

@@ -185,6 +193,11 @@ border-left: 2px solid var(--state-warning); padding-left: 12px; } + .policy-link { + display: inline-block; + margin-top: 10px; + color: var(--accent-primary); + } li { margin: 8px 0; color: var(--text-secondary); diff --git a/src/lib/ui/components/metrics/MetricsSummary.svelte b/src/lib/ui/components/metrics/MetricsSummary.svelte index 00ae066..91c6738 100644 --- a/src/lib/ui/components/metrics/MetricsSummary.svelte +++ b/src/lib/ui/components/metrics/MetricsSummary.svelte @@ -1,19 +1,14 @@