Make metrics reflect autoscaling policies (Refs #137)
CI / checks (push) Successful in 4m49s
CI / checks (push) Successful in 4m49s
This commit is contained in:
@@ -60,7 +60,7 @@ snapshots naturally show stale observations when viewed against today's clock.
|
||||
The fixture contains 3 hosts, 12 canonical Black Ops I/II servers, and 500 match
|
||||
records: 494 historical and 6 live. It includes solo through four-player squads,
|
||||
unknown starting counts, enough players and matches to exercise pagination,
|
||||
rosters, seven days of utilization and capacity samples, and retained Activity
|
||||
rosters, seven days of occupancy samples with unknown player limits, and retained Activity
|
||||
from campaigns, betas, host operations, maintenance, recovery, and autoscaling.
|
||||
The fleet has 6 live matches, 2 empty lobbies, a preparing server, and 3 problems:
|
||||
a stopped server, a server not responding, and a server on the offline host.
|
||||
|
||||
@@ -73,7 +73,7 @@ The production observation period and live pilot have not been run as part of im
|
||||
|
||||
The page retains up to 31 days of decision evidence and observations, with the 30 most recent decision changes shown. Repeated unchanged evaluations update the decision's last-seen time instead of adding a row. Unresolved intent survives retention. Provisioning history retains the operation and policy backlink separately.
|
||||
|
||||
Measurement totals exclude collection gaps. Request-to-verified-readiness summaries include queue time and only completed creates; they do not establish browser visibility or successful joins. “Time with players” in Metrics can rise simply because idle lobbies leave its denominator. Use host CPU and memory measurements alongside player-hours and spare availability when assessing savings.
|
||||
Measurement totals exclude collection gaps. Request-to-verified-readiness summaries include queue time and only completed creates; they do not establish browser visibility or successful joins. [Metrics](metrics.md) shows average and peak lobby counts, empty spare hours, and map time with no reported empty lobby. Its autoscaling signals compare historical counts with current policy targets. Use host CPU and memory measurements alongside player-hours and spare availability when assessing savings.
|
||||
|
||||
## API and implementation
|
||||
|
||||
|
||||
+108
-69
@@ -5,98 +5,137 @@ Choose the past 24 hours, 7 days, or 30 days. The page refreshes every minute.
|
||||
|
||||
## Overview and filters
|
||||
|
||||
The overview shows **Time with players** (occupied server-hours divided by known
|
||||
server-hours) and **Player-hours**. Known and unknown status time remain visible
|
||||
alongside the totals. **Last collected** is the newest recorded batch, including
|
||||
unknown checks; **Report refreshed** is when this report was calculated. Collection
|
||||
stops when no servers are registered. Earliest retained observations do not prove
|
||||
continuous history or the original installation date.
|
||||
**Lobbies running** shows the average and peak number of recorded lobbies in the
|
||||
selected scope. The average divides captured lobby-hours, including unknown
|
||||
status, by captured wall time. Empty spares are deliberate supply for new groups:
|
||||
the card shows their hours and share of all captured lobby-hours, rather than
|
||||
treating them as failed utilization. Unknown status can include lobbies starting
|
||||
or unavailable, so this count does not guarantee every lobby was ready to join.
|
||||
|
||||
**Player-hours** integrates reported player counts over time. **No empty lobby**
|
||||
shows the share of known map time when no tracked lobby reported zero players.
|
||||
Each game/map is measured separately, then its durations are pooled. This is not
|
||||
elapsed fleet time or a count of rejected joins. A partly occupied Zombies lobby
|
||||
does not count as an empty spare, even if player slots remain.
|
||||
|
||||
**Last collected** is the newest recorded batch, including unknown checks;
|
||||
**Report refreshed** is when this report was calculated. Collection stops when no
|
||||
servers are registered. Earliest retained observations do not prove continuous
|
||||
history or the original installation date.
|
||||
|
||||
Background refresh keeps the last successful report visible on failure and offers
|
||||
Retry. Returning to a visible tab refreshes immediately. The selected range is
|
||||
stored in the URL, and refreshes preserve table controls and chart selection.
|
||||
|
||||
Game and map filters apply to the whole report, including its comparisons,
|
||||
recommendations, and heatmap. Map choices preserve the exact recorded game/map
|
||||
pair. Range and scope are shareable through the URL. Table searches remain local.
|
||||
Retry. Returning to a visible tab refreshes immediately. Game and map filters apply
|
||||
to the whole report, including comparisons, signals, and the heatmap. Map choices
|
||||
preserve the exact recorded game/map pair. Range and scope are shareable through
|
||||
the URL; refreshes preserve table controls and chart selection.
|
||||
|
||||
## Timeline and tables
|
||||
|
||||
The timeline defaults to **Share of captured time**: each stack includes occupied,
|
||||
idle, and unknown server time, normalized to 100%. This denominator differs from the
|
||||
headline utilization percentage, which excludes unknown time. Switch to **Hours**
|
||||
to compare absolute server time. Periods without captured time remain gaps.
|
||||
The timeline has labelled axes and marks partial edge periods. Select a period,
|
||||
use Previous/Next, or select a bar to inspect its exact interval and values. The
|
||||
time table supplies the same figures for comparison and assistive technology.
|
||||
Zero captured time is distinct from an observed empty server. Short durations use
|
||||
seconds or minutes so a real observation is not rounded to zero.
|
||||
The timeline defaults to **Average lobbies running**, split into **With players**,
|
||||
**Empty spare**, and **Unknown**. Each bucket divides those lobby-hours by its
|
||||
captured wall time, so changing lobby counts remain visible. Collection gaps are
|
||||
excluded from that denominator. Switch to the share view to compare proportions,
|
||||
or hours to compare absolute lobby time. Periods without captured time remain gaps.
|
||||
|
||||
Map and server tables support numeric sorting and local search. Their searches
|
||||
apply only to the named table. Known-capacity duration stays next to all-full time.
|
||||
Server identities retain the technical name; rows describe history grouped by
|
||||
name, which may include different instances if a name was reused. **Open current
|
||||
server** appears only when that name is currently registered; its destination
|
||||
represents the current registration rather than every instance in the history.
|
||||
The timeline has labelled axes and marks partial edge periods. Select a bar or use
|
||||
Previous/Next to inspect its exact interval and values. The time table supplies
|
||||
the same figures for comparison and assistive technology. Zero captured time is
|
||||
distinct from an observed empty lobby. Short durations use seconds or minutes so
|
||||
a real observation is not rounded to zero.
|
||||
|
||||
## Comparisons and hosting experiments
|
||||
Map and instance tables support numeric sorting and local search. The map table's
|
||||
**No empty lobby** column uses occupancy alone and retains the known map-time
|
||||
denominator in its details. Unknown map availability stays distinct from zero
|
||||
time without a spare. Evidence can be limited by short observations or unknown
|
||||
occupancy, never by missing player limits.
|
||||
|
||||
**Explore this period** adds experimental decision aids below the main timeline:
|
||||
Instance rows describe history grouped by server name, which can include different
|
||||
instances if a name was reused. **Open current server** appears only when that name
|
||||
is currently registered; its destination represents the current registration.
|
||||
|
||||
- **Map capacity pressure** pools all-full map-hours divided by known-capacity
|
||||
map-hours across the selected map groups. Overlapping map intervals are counted
|
||||
separately. The score is not elapsed fleet time or a fleet health score; its
|
||||
denominator and contributing map count remain visible.
|
||||
- **Previous-period comparisons** use the preceding equally long window with
|
||||
identical filters. Utilization changes are percentage points; player-hours
|
||||
changes are relative percentages. A previous observed zero can show **New**.
|
||||
Missing evidence remains unavailable. Partial retained history is labelled;
|
||||
a 30-day comparison usually has little previous history because default retention is
|
||||
31 days. Both periods' known time is shown. Bounds cannot establish that
|
||||
collection was uninterrupted, and changes are not adjusted for fleet size.
|
||||
- **Activity heatmap** groups captured time by weekday and hour in its displayed
|
||||
timezone. Color represents occupied / known server time. Exact cell values
|
||||
include unknown time and contributing dates. No-data cells differ from observed
|
||||
zero utilization; repeated DST hours contribute their actual elapsed time.
|
||||
- **Hosting experiments** are generated automatically from the selected map
|
||||
observations. They are advisory and never create, restart, or destroy a server.
|
||||
## Comparisons and activity
|
||||
|
||||
Hosting Rules v1 deliberately offer experiments even with small samples:
|
||||
Previous-period comparisons use the preceding equally long window with identical
|
||||
filters. Average lobby counts compare recorded supply, while player-hours changes
|
||||
are relative percentages. A previous observed zero can show **New**. Missing
|
||||
evidence remains unavailable. Partial retained history is labelled; a 30-day
|
||||
comparison usually has little previous history because default retention is
|
||||
31 days. Collection gaps are excluded, and comparisons do not prove uninterrupted
|
||||
collection or adjust player-hours for fleet size.
|
||||
|
||||
| Recorded condition | Suggested experiment |
|
||||
| ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- |
|
||||
| At least 20% of known capacity time had every lobby full | Trial one additional lobby during busy hours. |
|
||||
| Capacity unavailable and players present for at least 80% of known server time | Tentatively trial an extra lobby using occupancy as a proxy. |
|
||||
| Players present for at most 10% of known time, with under 1% measured pressure or unavailable capacity | Review one idle lobby or a shorter hosting window while preserving access to the map. |
|
||||
| Occupancy unavailable | Check telemetry/RCON reporting and watch for known player counts. |
|
||||
| Other observations | Check missing capacity or keep the lineup and watch peaks. |
|
||||
The activity heatmap groups captured time by weekday and hour in its displayed
|
||||
timezone. Color represents occupied / known lobby time. Exact cell values include
|
||||
unknown time and contributing dates. No-data cells differ from observed zero;
|
||||
repeated DST hours contribute their actual elapsed time. Occupancy share describes
|
||||
how the fleet was used, but can change simply because a policy adds or removes
|
||||
spares. Read it alongside player-hours, lobby counts, and empty-lobby availability.
|
||||
|
||||
Each suggestion includes evidence and an outcome to watch. Unknown capacity,
|
||||
less than one hour of relevant known evidence, or substantial unknown captured
|
||||
status makes the suggestion tentative. These thresholds are product hypotheses;
|
||||
occupancy does not establish failed joins or a need for additional capacity.
|
||||
Suggestions use historical observations. Check the current lineup before trying
|
||||
a change; a retained map may no longer have a hosted lobby.
|
||||
## Autoscaling signals
|
||||
|
||||
Signals compare each map's observations with its **current** applicable policy.
|
||||
Game-wide policies apply independently to each supported map variant. Each signal
|
||||
links to the policy on **Fleet → Autoscaling**; a map without a policy gets a link
|
||||
to add one. Signals are advisory and never change a policy or a lobby.
|
||||
|
||||
| Recorded condition | Suggested action |
|
||||
| ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
|
||||
| No empty lobby for at least 2% of known map time, mostly at or above the current maximum | Review raising the policy maximum. |
|
||||
| No empty lobby for at least 2% of known map time, mostly below the current maximum | Review the spare target and check creation time or blockers. |
|
||||
| An applicable policy is off or paused | Review the policy and its pause or activation conditions. |
|
||||
| No applicable policy and known occupancy is available | Consider adding an autoscaling policy to keep spares available. |
|
||||
| Occupancy unavailable | Check telemetry/RCON reporting and wait for known player counts. |
|
||||
| No shortage signal | Keep watching empty-lobby availability. |
|
||||
|
||||
Evidence includes no-empty map time and how it splits above/below the current
|
||||
maximum. Less than one hour of known map availability, less than 80% known map
|
||||
availability, or less than 80% known lobby time makes advice tentative. Player limits are not needed. These
|
||||
thresholds are advisory rules, not demand forecasts.
|
||||
|
||||
Policy settings may have changed during the selected period. Comparing historical
|
||||
counts against today's maximum does not establish which settings applied then or
|
||||
whether autoscaling was enabled. No reported empty lobby also does not prove
|
||||
failed joins or unmet demand. Review provisioning delays, blockers, host resources,
|
||||
and game-key availability before changing targets.
|
||||
|
||||
## Definitions and collection
|
||||
|
||||
- Utilized means a successful status response reported at least one player.
|
||||
- Idle means a successful response reported zero players.
|
||||
- Unknown covers failed checks, provisioning, and unavailable containers; it is excluded from utilization percentages.
|
||||
- Player-hours integrate the reported player count over observed time. Bots are included if the server counts them as players.
|
||||
- All lobbies full measures map wall time when every tracked lobby in that game/map group has a known capacity and is full. Missing capacity or status makes that interval unknown. This is a capacity-pressure indicator, not a count of rejected joins or unmet demand.
|
||||
- **With players:** a successful observation reported at least one player.
|
||||
- **Empty spare:** an observation reported zero players.
|
||||
- **Unknown:** failed checks, provisioning, or otherwise unavailable occupancy.
|
||||
- **Player-hours:** reported players × observed time. Bots count if the server includes them.
|
||||
- **Empty lobby available:** at least one lobby on the game/map reported zero players.
|
||||
- **No empty lobby:** at least one lobby reported occupancy, and none reported zero. Mixed positive and unknown counts mean no empty lobby was reported; an unknown lobby might actually have been empty.
|
||||
- **Unknown map availability:** every lobby on the game/map had unknown occupancy. This interval is excluded from the known-map denominator and is never counted as a spare shortage.
|
||||
|
||||
Observations are recorded by the existing status worker. Each remains valid until the next collection, up to the 90-second freshness limit. Longer manager outages are excluded. Charts split time at UTC hour/day boundaries; partial periods remain partial. Map variants use the configured variant when it matches the reported map.
|
||||
Observations come from the existing status worker. Each complete collection stays
|
||||
valid until the next collection, up to the 90-second freshness limit. Longer gaps
|
||||
are excluded, not counted as unknown or zero lobbies. Captured wall time includes
|
||||
intervals with at least one recorded lobby in the selected scope. Unknown lobby
|
||||
time and unknown map time use different units: several unknown lobbies on one map
|
||||
contribute several lobby-hours but only one interval of unknown map availability.
|
||||
|
||||
Capacity comes from `sv_maxclients` / `sv_maxClients` in the status response. When unavailable, the UI displays unknown rather than assuming a four-player limit. Capacity coverage is the map time for which every tracked lobby reported both occupancy and capacity.
|
||||
Charts split time at UTC hour/day boundaries; partial periods remain partial.
|
||||
Map variants use the configured variant when it matches the reported map. Player
|
||||
capacity may be retained in raw samples, but no metric or signal requires
|
||||
`sv_maxclients` or an assumed four-player limit.
|
||||
|
||||
Collection begins after this feature is installed; previous occupancy cannot be reconstructed. Samples default to 31 days of retention; Admin → Settings can change this period. Cleanup runs during collection and in the background. They survive removal of a server and contain no player identities. The additive database table is created automatically on startup.
|
||||
Collection begins after this feature is installed; earlier occupancy cannot be
|
||||
reconstructed. Samples default to 31 days of retention; **Admin → Settings** can
|
||||
change this period. Cleanup runs during collection and in the background. Samples
|
||||
survive removal of a server and contain no player identities.
|
||||
|
||||
## API
|
||||
|
||||
`GET /api/metrics` supplies the same report as the page and disables response
|
||||
`GET /api/metrics` supplies the same report as the page and disables HTTP response
|
||||
caching. Query parameters are `days` (1, 7, or 30), `game`, `map` (requires game),
|
||||
and `timeZone` (an IANA zone). Invalid ranges and zones fall back to the defaults;
|
||||
invalid scope syntax is normalized. Fleet collection timestamps remain separate
|
||||
from selected-scope evidence timestamps. Collection boundaries are determined
|
||||
before scope filtering, so samples from other groups still end a batch interval.
|
||||
|
||||
`running` contains `average`, `peak`, and `capturedHours`; timeline points include
|
||||
their own `capturedHours`. Map rows expose `noEmpty`, `availabilityObserved`, and
|
||||
`availabilityUnknown` durations, plus current `policy` context and
|
||||
`noEmptyAtMaximum` / `noEmptyBelowMaximum` durations. `pooledAvailability` combines
|
||||
map durations and coverage. Policy edits and global pause changes invalidate
|
||||
cached metrics reports.
|
||||
|
||||
Reference in New Issue
Block a user