Relax freshness limits and tolerate transient RCON failures

This commit is contained in:
2026-10-05 16:36:23 -04:00
parent d8b972cfc3
commit b1b3e8b958
28 changed files with 221 additions and 58 deletions
+10 -1
View File
@@ -102,9 +102,18 @@ reachable. Finished scheduled and legacy maintenance activity is retained for
## Health recovery and configuration drift
Fleet keeps successful game and running-container reports fresh for 90 seconds.
Two consecutive failed monitoring checks mark RCON health stale; a first timeout
or send failure remains visible in diagnostics without changing the health badge.
Configuration and invalid-reply errors, including authentication failures, take
effect immediately, as do failed manual probes. Successful RCON checks reset the
failure count; skipped checks do not count. Without fresh telemetry, RCON evidence
older than 120 seconds is offline. Explicit stopped/unhealthy container states
and lifecycle changes still take precedence.
The status monitor collects container and game health every 30 seconds. Once per
minute, the watchdog evaluates that completed evidence without issuing its own
routine health probes. Samples expire after 45 seconds; missing or stale evidence
routine health probes. Samples expire after 90 seconds; missing or stale evidence
leaves health unknown and cannot justify recovery. Samples retain their original
timestamps and instance identity, and each failure sample counts at most once.
A monitoring gap or changed instance resets the sustained failure window. After
+1 -1
View File
@@ -78,7 +78,7 @@ a change; a retained map may no longer have a hosted lobby.
- Player-hours integrate the reported player count over observed time. Bots are included if the server counts them as players.
- All lobbies full measures map wall time when every tracked lobby in that game/map group has a known capacity and is full. Missing capacity or status makes that interval unknown. This is a capacity-pressure indicator, not a count of rejected joins or unmet demand.
Observations are recorded by the existing status worker. Each remains valid until the next collection, up to the 45-second freshness limit. Longer manager outages are excluded. Charts split time at UTC hour/day boundaries; partial periods remain partial. Map variants use the configured variant when it matches the reported map.
Observations are recorded by the existing status worker. Each remains valid until the next collection, up to the 90-second freshness limit. Longer manager outages are excluded. Charts split time at UTC hour/day boundaries; partial periods remain partial. Map variants use the configured variant when it matches the reported map.
Capacity comes from `sv_maxclients` / `sv_maxClients` in the status response. When unavailable, the UI displays unknown rather than assuming a four-player limit. Capacity coverage is the map time for which every tracked lobby reported both occupancy and capacity.
+1 -1
View File
@@ -57,7 +57,7 @@ unreachable Komodo API is not evidence that a stop succeeded.
Disruptive operations require a fresh zero-player result from RCON or accepted
telemetry, unless Komodo positively reports a stopped container. Fresh occupied
telemetry vetoes a conflicting empty RCON reading. Reports must be received within
45 seconds; recovery verification also excludes reports from before the submitted
90 seconds; recovery verification also excludes reports from before the submitted
operation. Telemetry can verify the game's activity and selected map after Komodo
reports healthy. Runtime actions wait for conflicting
campaign, beta, maintenance, or provisioning work. Duplicate pending requests for