Make metrics reflect autoscaling policies (Refs #137)
CI / checks (push) Successful in 4m49s

This commit is contained in:
2026-10-10 14:48:52 -04:00
parent bd34c3c9e7
commit 33da4482a6
24 changed files with 1264 additions and 370 deletions
+1 -1
View File
@@ -73,7 +73,7 @@ The production observation period and live pilot have not been run as part of im
The page retains up to 31 days of decision evidence and observations, with the 30 most recent decision changes shown. Repeated unchanged evaluations update the decision's last-seen time instead of adding a row. Unresolved intent survives retention. Provisioning history retains the operation and policy backlink separately.
Measurement totals exclude collection gaps. Request-to-verified-readiness summaries include queue time and only completed creates; they do not establish browser visibility or successful joins. “Time with players” in Metrics can rise simply because idle lobbies leave its denominator. Use host CPU and memory measurements alongside player-hours and spare availability when assessing savings.
Measurement totals exclude collection gaps. Request-to-verified-readiness summaries include queue time and only completed creates; they do not establish browser visibility or successful joins. [Metrics](metrics.md) shows average and peak lobby counts, empty spare hours, and map time with no reported empty lobby. Its autoscaling signals compare historical counts with current policy targets. Use host CPU and memory measurements alongside player-hours and spare availability when assessing savings.
## API and implementation
+108 -69
View File
@@ -5,98 +5,137 @@ Choose the past 24 hours, 7 days, or 30 days. The page refreshes every minute.
## Overview and filters
The overview shows **Time with players** (occupied server-hours divided by known
server-hours) and **Player-hours**. Known and unknown status time remain visible
alongside the totals. **Last collected** is the newest recorded batch, including
unknown checks; **Report refreshed** is when this report was calculated. Collection
stops when no servers are registered. Earliest retained observations do not prove
continuous history or the original installation date.
**Lobbies running** shows the average and peak number of recorded lobbies in the
selected scope. The average divides captured lobby-hours, including unknown
status, by captured wall time. Empty spares are deliberate supply for new groups:
the card shows their hours and share of all captured lobby-hours, rather than
treating them as failed utilization. Unknown status can include lobbies starting
or unavailable, so this count does not guarantee every lobby was ready to join.
**Player-hours** integrates reported player counts over time. **No empty lobby**
shows the share of known map time when no tracked lobby reported zero players.
Each game/map is measured separately, then its durations are pooled. This is not
elapsed fleet time or a count of rejected joins. A partly occupied Zombies lobby
does not count as an empty spare, even if player slots remain.
**Last collected** is the newest recorded batch, including unknown checks;
**Report refreshed** is when this report was calculated. Collection stops when no
servers are registered. Earliest retained observations do not prove continuous
history or the original installation date.
Background refresh keeps the last successful report visible on failure and offers
Retry. Returning to a visible tab refreshes immediately. The selected range is
stored in the URL, and refreshes preserve table controls and chart selection.
Game and map filters apply to the whole report, including its comparisons,
recommendations, and heatmap. Map choices preserve the exact recorded game/map
pair. Range and scope are shareable through the URL. Table searches remain local.
Retry. Returning to a visible tab refreshes immediately. Game and map filters apply
to the whole report, including comparisons, signals, and the heatmap. Map choices
preserve the exact recorded game/map pair. Range and scope are shareable through
the URL; refreshes preserve table controls and chart selection.
## Timeline and tables
The timeline defaults to **Share of captured time**: each stack includes occupied,
idle, and unknown server time, normalized to 100%. This denominator differs from the
headline utilization percentage, which excludes unknown time. Switch to **Hours**
to compare absolute server time. Periods without captured time remain gaps.
The timeline has labelled axes and marks partial edge periods. Select a period,
use Previous/Next, or select a bar to inspect its exact interval and values. The
time table supplies the same figures for comparison and assistive technology.
Zero captured time is distinct from an observed empty server. Short durations use
seconds or minutes so a real observation is not rounded to zero.
The timeline defaults to **Average lobbies running**, split into **With players**,
**Empty spare**, and **Unknown**. Each bucket divides those lobby-hours by its
captured wall time, so changing lobby counts remain visible. Collection gaps are
excluded from that denominator. Switch to the share view to compare proportions,
or hours to compare absolute lobby time. Periods without captured time remain gaps.
Map and server tables support numeric sorting and local search. Their searches
apply only to the named table. Known-capacity duration stays next to all-full time.
Server identities retain the technical name; rows describe history grouped by
name, which may include different instances if a name was reused. **Open current
server** appears only when that name is currently registered; its destination
represents the current registration rather than every instance in the history.
The timeline has labelled axes and marks partial edge periods. Select a bar or use
Previous/Next to inspect its exact interval and values. The time table supplies
the same figures for comparison and assistive technology. Zero captured time is
distinct from an observed empty lobby. Short durations use seconds or minutes so
a real observation is not rounded to zero.
## Comparisons and hosting experiments
Map and instance tables support numeric sorting and local search. The map table's
**No empty lobby** column uses occupancy alone and retains the known map-time
denominator in its details. Unknown map availability stays distinct from zero
time without a spare. Evidence can be limited by short observations or unknown
occupancy, never by missing player limits.
**Explore this period** adds experimental decision aids below the main timeline:
Instance rows describe history grouped by server name, which can include different
instances if a name was reused. **Open current server** appears only when that name
is currently registered; its destination represents the current registration.
- **Map capacity pressure** pools all-full map-hours divided by known-capacity
map-hours across the selected map groups. Overlapping map intervals are counted
separately. The score is not elapsed fleet time or a fleet health score; its
denominator and contributing map count remain visible.
- **Previous-period comparisons** use the preceding equally long window with
identical filters. Utilization changes are percentage points; player-hours
changes are relative percentages. A previous observed zero can show **New**.
Missing evidence remains unavailable. Partial retained history is labelled;
a 30-day comparison usually has little previous history because default retention is
31 days. Both periods' known time is shown. Bounds cannot establish that
collection was uninterrupted, and changes are not adjusted for fleet size.
- **Activity heatmap** groups captured time by weekday and hour in its displayed
timezone. Color represents occupied / known server time. Exact cell values
include unknown time and contributing dates. No-data cells differ from observed
zero utilization; repeated DST hours contribute their actual elapsed time.
- **Hosting experiments** are generated automatically from the selected map
observations. They are advisory and never create, restart, or destroy a server.
## Comparisons and activity
Hosting Rules v1 deliberately offer experiments even with small samples:
Previous-period comparisons use the preceding equally long window with identical
filters. Average lobby counts compare recorded supply, while player-hours changes
are relative percentages. A previous observed zero can show **New**. Missing
evidence remains unavailable. Partial retained history is labelled; a 30-day
comparison usually has little previous history because default retention is
31 days. Collection gaps are excluded, and comparisons do not prove uninterrupted
collection or adjust player-hours for fleet size.
| Recorded condition | Suggested experiment |
| ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- |
| At least 20% of known capacity time had every lobby full | Trial one additional lobby during busy hours. |
| Capacity unavailable and players present for at least 80% of known server time | Tentatively trial an extra lobby using occupancy as a proxy. |
| Players present for at most 10% of known time, with under 1% measured pressure or unavailable capacity | Review one idle lobby or a shorter hosting window while preserving access to the map. |
| Occupancy unavailable | Check telemetry/RCON reporting and watch for known player counts. |
| Other observations | Check missing capacity or keep the lineup and watch peaks. |
The activity heatmap groups captured time by weekday and hour in its displayed
timezone. Color represents occupied / known lobby time. Exact cell values include
unknown time and contributing dates. No-data cells differ from observed zero;
repeated DST hours contribute their actual elapsed time. Occupancy share describes
how the fleet was used, but can change simply because a policy adds or removes
spares. Read it alongside player-hours, lobby counts, and empty-lobby availability.
Each suggestion includes evidence and an outcome to watch. Unknown capacity,
less than one hour of relevant known evidence, or substantial unknown captured
status makes the suggestion tentative. These thresholds are product hypotheses;
occupancy does not establish failed joins or a need for additional capacity.
Suggestions use historical observations. Check the current lineup before trying
a change; a retained map may no longer have a hosted lobby.
## Autoscaling signals
Signals compare each map's observations with its **current** applicable policy.
Game-wide policies apply independently to each supported map variant. Each signal
links to the policy on **Fleet → Autoscaling**; a map without a policy gets a link
to add one. Signals are advisory and never change a policy or a lobby.
| Recorded condition | Suggested action |
| ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| No empty lobby for at least 2% of known map time, mostly at or above the current maximum | Review raising the policy maximum. |
| No empty lobby for at least 2% of known map time, mostly below the current maximum | Review the spare target and check creation time or blockers. |
| An applicable policy is off or paused | Review the policy and its pause or activation conditions. |
| No applicable policy and known occupancy is available | Consider adding an autoscaling policy to keep spares available. |
| Occupancy unavailable | Check telemetry/RCON reporting and wait for known player counts. |
| No shortage signal | Keep watching empty-lobby availability. |
Evidence includes no-empty map time and how it splits above/below the current
maximum. Less than one hour of known map availability, less than 80% known map
availability, or less than 80% known lobby time makes advice tentative. Player limits are not needed. These
thresholds are advisory rules, not demand forecasts.
Policy settings may have changed during the selected period. Comparing historical
counts against today's maximum does not establish which settings applied then or
whether autoscaling was enabled. No reported empty lobby also does not prove
failed joins or unmet demand. Review provisioning delays, blockers, host resources,
and game-key availability before changing targets.
## Definitions and collection
- Utilized means a successful status response reported at least one player.
- Idle means a successful response reported zero players.
- Unknown covers failed checks, provisioning, and unavailable containers; it is excluded from utilization percentages.
- Player-hours integrate the reported player count over observed time. Bots are included if the server counts them as players.
- All lobbies full measures map wall time when every tracked lobby in that game/map group has a known capacity and is full. Missing capacity or status makes that interval unknown. This is a capacity-pressure indicator, not a count of rejected joins or unmet demand.
- **With players:** a successful observation reported at least one player.
- **Empty spare:** an observation reported zero players.
- **Unknown:** failed checks, provisioning, or otherwise unavailable occupancy.
- **Player-hours:** reported players × observed time. Bots count if the server includes them.
- **Empty lobby available:** at least one lobby on the game/map reported zero players.
- **No empty lobby:** at least one lobby reported occupancy, and none reported zero. Mixed positive and unknown counts mean no empty lobby was reported; an unknown lobby might actually have been empty.
- **Unknown map availability:** every lobby on the game/map had unknown occupancy. This interval is excluded from the known-map denominator and is never counted as a spare shortage.
Observations are recorded by the existing status worker. Each remains valid until the next collection, up to the 90-second freshness limit. Longer manager outages are excluded. Charts split time at UTC hour/day boundaries; partial periods remain partial. Map variants use the configured variant when it matches the reported map.
Observations come from the existing status worker. Each complete collection stays
valid until the next collection, up to the 90-second freshness limit. Longer gaps
are excluded, not counted as unknown or zero lobbies. Captured wall time includes
intervals with at least one recorded lobby in the selected scope. Unknown lobby
time and unknown map time use different units: several unknown lobbies on one map
contribute several lobby-hours but only one interval of unknown map availability.
Capacity comes from `sv_maxclients` / `sv_maxClients` in the status response. When unavailable, the UI displays unknown rather than assuming a four-player limit. Capacity coverage is the map time for which every tracked lobby reported both occupancy and capacity.
Charts split time at UTC hour/day boundaries; partial periods remain partial.
Map variants use the configured variant when it matches the reported map. Player
capacity may be retained in raw samples, but no metric or signal requires
`sv_maxclients` or an assumed four-player limit.
Collection begins after this feature is installed; previous occupancy cannot be reconstructed. Samples default to 31 days of retention; Admin → Settings can change this period. Cleanup runs during collection and in the background. They survive removal of a server and contain no player identities. The additive database table is created automatically on startup.
Collection begins after this feature is installed; earlier occupancy cannot be
reconstructed. Samples default to 31 days of retention; **Admin → Settings** can
change this period. Cleanup runs during collection and in the background. Samples
survive removal of a server and contain no player identities.
## API
`GET /api/metrics` supplies the same report as the page and disables response
`GET /api/metrics` supplies the same report as the page and disables HTTP response
caching. Query parameters are `days` (1, 7, or 30), `game`, `map` (requires game),
and `timeZone` (an IANA zone). Invalid ranges and zones fall back to the defaults;
invalid scope syntax is normalized. Fleet collection timestamps remain separate
from selected-scope evidence timestamps. Collection boundaries are determined
before scope filtering, so samples from other groups still end a batch interval.
`running` contains `average`, `peak`, and `capturedHours`; timeline points include
their own `capturedHours`. Map rows expose `noEmpty`, `availabilityObserved`, and
`availabilityUnknown` durations, plus current `policy` context and
`noEmptyAtMaximum` / `noEmptyBelowMaximum` durations. `pooledAvailability` combines
map durations and coverage. Policy edits and global pause changes invalidate
cached metrics reports.